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PRKFA<^t: 


One  <)l  the  major  findings  in  the  A(iARD  Might  Mechanics  Panel  (FMl*)  ACiARl>t)graph  No.27d.  ’Survey  of  Missile 
Simulation  and  Flight  Mechanics  Facilities  in  NA  IO'*.  by  Willard  M. Holmes,  was  the  fact  that.  ’Nolably  missing  from  a  vast 
majority  of  facilities  were  any  established  or  formal  procedures  for  accomplishing  any  level  of  simulation  model  validation." 
The  A(iARI)  FMP  felt  this  omission  warranted  further  action  and  subsetiuently  reetunmended  the  formation  ol  Working 
Group  1 2.  "Validation  of  Missile  System  Simulation". 

Once  approved  by  the  ACiARl)  National  Delegates.  Working  (iroup  12  held  huir  meetings  over  the  1^X2—  1  dS4  time 
period  as  follows:  (a)  2 1  October  IVS2,  London.  F!ngland:(b)  5—6  May  IdK.^.  Miinehen.  West  Germany;  (c)  24—26 
October  I  yS3.  Fglin  AFB  FL.  USA.  and  finally  (d)  27— 2^  March  I  *>84.  Paris.  France. 

This  report  documents  the  working  group  findings.  The  working  group  consisting  of  Mr  M.J. Donat.  France:  Mr  Werner 
Bub,  Germany;  Mr  Klaus  flausel.  Germany;  Mr  Karl  F-rnst  Platt,  (iermany:  Mr  Amileare  (ia^/ina.  Italy:  Mr  B.J.Wanstall. 
United  Kingdom:  Mr  D  Hyde.  United  Kingdom:  Dr  Willard  llt)lmes.  United  .Stales;  Mr  Lawrence  Byrd.  United  Slates 
(representing  the  AGARD  (iuidanceand  Gontrol  Panel),  and  Mr  Ronald  Anderson.  L’nited  Stales. 


RONALDO  ANDI  RSON 
FMP  Member 
Chairman.  W{i- 1  2 
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1.  INTRODUCTION 

Tbe  AGAP.D  Flight  Mechanics?  Panel  (FMP)  has  been  concerned  with  the 
problem  of  missile  simulation  in  general,  and  cooperative  NATO 
developments  of  missile  system  simulations,  test,  and  evaluation  as 
related  to  missile  system  flight  mechanics  In  particular.  As  a  result, 
the  FMP  sponsored  a  survey  of  missile  simulation  facilities  reported  in 
AGAPPograph  No.  279,  "Survey  of  Missile  Simulation  and  Flight  Mechanics 
in  NATO,"  by  Willard  M.  Holmes,  April  19P3.  One  of  the  key  findings  in 
this  survey  was  that  very  little  effort  Is  being  expended  in  missile 
simulation  validation.  Furthermore,  validation  techniaues  themselves 
are  generallv  not  standard;  often  being  ill-defined  or  undocumented. 

This  fact  motivated  the  formation  of  FMP  Working  Group  12, 
"Validation  of  Missile  Svsten  Simulation."  This  report  documents  the 
group  findings. 

Although  there  Is  no  shortage  of  reference  material  on  missile 
simulations  per  se,  there  seems  to  he  no  "history"  on  the  subiect,  1  ike- 
the  manned  simulation  community,  le.g..  The  Pilot  Maker,  by  Floyd  T.. 
Kelly  as  told  to  Robert  R.  Parks,  Grosser  and  Dunlap,  1970).  The  manned 
simulation  world,  founded  first  in  training  simulators  and  later  in 
research  anc  development  simulators,  is  also  lust  now  realizing  the  need 
for  simulation  validation. 

Perhaps,  In  both  cases,  the  evolution  of  simulation  equipment  from 
extremely  crude  mechanical  devices  to  modern  electronic  "arcade  game 
simulators"  happened  so  fast  that  It  Is  only  recently  that  people  began 
to  ask,  "Po  they  really  work"?  Or,  perhaps  the  early  equipment 
reliability  problems  were  snob  that  everyone  was  quite  happy  If  the 
simulation  appeared  to  be  "working.",  let  alone  providing  "useful" 
results . 

In  any  event,  times  have  changed,  and  in  both  manned  aircraft 
simulations  and  missile  simulations  the  question  of  validation  is 
becoming  of  more  and  more  serious  concern. 

But  where  does  ore  start  in  terms  of  defining  how  to  validate 
missile  simulations  when  few  organizations  do  any  formal  validation  now 
and  there  are  no  agreed  upon  standards  whatsoever?  This  was  the 
extremely  broad  and  difficult  question  the  working  group  had  to 
repeatedly  ask  itself. 

After  a  number  of  false  starts,  the  working  group  concluded  that 
the  most  Important  objective  was  to  find  a  "method"  o^  organizing 
simulation  validation  techniques  rather  than  a  collection  of  actual 
methods.  In  addition,  the  concept  of  "confidence"  ii'i  missile 
simulations  began  to  play  a  dominant  role  in  tbe  group  thinking  from  the 
^’ory  start  of  the  activity.  That  is,  the  purpose  o^  the  validation  was 
as  important  as,  or  perhaps  more  Important  than,  the  methods  themselves. 
In  addition,  m.ariy  the  thoughts  on  validation  methods  .seemed  much 
easier  to  organize  if  one  thought  in  terms  of  developlr.g  confidence  in 
the  behavior  of  a  simulated  model.  Therefore,  a  perbaps  larger  portion 
of  the  group  effort  w'as  expended  on  a  hierarchical  model  representation 
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of  Confidence  Level  in  Model 
tlon  that  is  integral  to  the 
final  result.  In  fact,  this 
than  originally  envisioned, 
of  the  major  portions  of  the 


Behavior  (CLIMB)  that  included  documenta- 
total  process  to  insure  "confidence"  in  the 
documentation  may  Instill  more  confidence 
This  "organization  of  thought"  provides  one 
group  final  results. 


The  remainder  of  this  report  is  organized  as  follows.  A 
Terminology  Section  is  provided  to  Include  some  fundamental  definitions. 
LTiile  not  intended  to  be  "final",  this  section  presents  a  good  start  on 
a  standard  set  of  terms.  The  next  section  on  the  CLIMB  process  forms 
the  heart  of  this  report  and  is  by  far  the  most  Important  contribution 
the  group  could  provide  within  the  time  span  available.  Hopefully,  this 
section  will  form  a  basis  for  further  work  in  the  area  plus  a 
stand-alone  start  toward  a  unified  approach  to  missile  system  simulation 
validation . 


The  next  section  on  Computer  Languages  could  also  be  a  separ.ate 
working  group  topic,  being  only  touched  upon  herd. 

Next,  a  brief  section  on  Software  Valldation/Verlfi cation/ 
Assessment  Methods  follows.  This  section  could  easily  be  the  founaation 
of  a  working  group  report  by  itself,  but  this  task  was  clearly  beyond 
the  two-year  effort  of  the  present  activity. 

Finally,  overall  recommendations  are  presented.  As  might  be 
expected,  the  extremely  difficult  subject  has  led  to  a  nun, her  of 
suggested  follow-on  activities  for  FMP  consideration. 

Throughout  the  v?orking  group  activity,  the  members  v;ere  extremely 
conscientious  and  devoted  to  their  task.  Each  meeting  generated  a  mound 
of  documentation  to  be  shared  by  the  other  members  prior  to  the  next 
meeting.  These  notes,  or  "homework",  are  too  voluminous  to  reproduce 
here,  however,  the  information  was  invaluable  in  the  development  of  this 
final  report  and  in  forming  the  thoughts  of  each  group  member.  The 
members  themselves  represented  one  of  the  most  professional  groups  c’nc- 
Chalrman  has  ever  been  associated  with,  and  hopefully  this  final  report 
reflects  this  fact.  If  not,  the  problem  was  the  sheer  complexity  atid 
difficulty  associated  with  the  working  group  task  and  not  the  quality 
and  enthusiasm  of  the  members  themselves. 
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2.1  IMF.OPUCTION 


II. 


irpMINOLOCY 


The  following'  general  ternlnclogy  was  adopted  by  the  working  group. 
This  terninolog'-  is  a  somewhat  modified  version  of  the  list  that  .tppear 
in  the  March  1979  issue  of  SIliULATTON; 

To  provide  a  proper  framework  to  review  the  credibility  of  a 
simulation,  it  is  convenient  to  divide  the  simulation  environment  into 
three  basic  elements  as  depicted  in  Figure  2-1.  The  inner  arrows 
describe  the  processes  which  relate  the  elements  to  each  other  and  the 
outer  arrows  refer  to  the  procedures  which  evaluate  the  creclhility  of 
these  processes.  This  basic  pictorial  concept  can  be  further  refined  as 
shown  in  Figure  2-2: 

Fach  of  the  basic  elements  and  their  interrelationships  are  dealt  with 
in  the  following  set  of  definltlors: 

2.2  PESCRIPTIPK  OF  TERMIMOLOGY 

MISSILE  SIMULATIOK  All  analvtical,  digital  computer, 

hvbrld  computer,  or 
hardware-ln-the-loop  dynamic 
evaluations  of  tactical  missiles  to 
pain  insight  about  reality 

MODEL  DCCUMENTATTOy  Svstematic  and  coherent  written  and 

graphical  representation  of  the 
associated  data  base  according  to  a 
specified  format  and  with  a  specific 
purpose 

REALTY''  An  entity,  situation,  or  system  which 

has  been  selected  for  analysis 

ori'CEPTUAL  MODE!,  Model  huilder's  perception  and 

understanding  ot  tbc  system  to  be 
simulated.  It  consists  of  a 
hypothetical  complete  explanation  of 
how  the  system  functions.  It  often 
rakes  the  form  of  verbal  description, 
complemented  by  block  diagrams,  flow 
charts  and  systems  specifications. 

In  most  cases,  the  large  complexity 
of  the  conceptual  model  precludes  its 
consideration  as  a  possible 

simulation  model.  In  view  of  the 

« 

requirements  of  the  intended 
simulation  studies,  the  modeler 
establishes  the  complexity  of  the 
simulation  model  and  degree  of  detail 
necessary.  This  information, 
generally  in  descriptive  (verbal) 
form,  complemented  by  block  diagrams 
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and  flow  charts  constitutes  the 
Conceptual  Simulation  Model  or 
abbreviated  to  Conceptual  Model.  At 
the  same  time,  this  represents  the 
requirenents  on  tlie  formal  simulation 
model 


The  formal  model  provides  the 
technical  description  of  the 
simulation  model.  Tt  takes  the  form 
of  mathematical  equations,  adequate 
description  of  logic  flow  and  model 
data,  complemented  bv  the  necessary 
detailed  text.  both  the  conceptual 
and  the  formal  model  together  form 
the  theoretical  model 


DOMAIN  or  INTENDFP  APPLICATION  Prescribed  conditions  for  which  the 

(or  CONCEPTUAL  MODEL)  CONCEPTUAL  MODEL  is  intended  to  march 

REALITY 


LEVEL  OF  agreement 
(OF  CONCEPTUAL  MODEL) 


MODEL  nUALl El CATION 


Expected  agreement  between  the 
CONCEPTIIAT.  MODEL  and  REALITY, 
consistent  with  the  DOMAIN  OF 
INTENDED  APPI ICATION  and  the  purpose 
for  which  the  model  was  built 

Determination  of  adequacy  of  the 
CONCEPTUAL  MODEL  to  provide  an 
acceptable  I.EVEL  OF  AGREEMENT  for  thr- 
DOMAIN  OF  INTENDED  APPLICATION 

This  may  involve  a  cemparisen  of 
alternate  methods  of  missile 
simulation  to  establish  credibility 
with  two  or  more  independent  data 
sets  or  results 


COMPUTERIZED  MODEL  An  operational  computer  program  which 

implements  a  CONCEPTUAL  MODEL 

MODEL  VERIFICATION  The  process  of  showing  that  tPf 

proposed  conceptvinl  and  the 
associated  formal  model  are  an 
adequate  and  consistent 
representation  of  the  system  to  be 
simulated,  all  in  view  of  the 
intended  application.  The  method 
used  is  basically  expert  critique, 
which  makes  use  of  expertise  and  past 
experience  in  order  to  assess  the 
adequacy  the  conceptual  model  and 
the  derivation  of  the  formal  model. 
Suitable  documentation  should  allow 


following  anil  understanding  the  Ideas 
of  the  model  builder  in  deriving  the 
theoretical  model 


DOMAIN  OF  APPLICABIbITY 
(OF  COMPUTERIZED  MODEL) 


RANGE  OF  ACCURACY 
(OF  COMPUTERIZED  MODEL) 

IMPLEMENTATION 


PROGRAM  VERIFICATION 


MODEL  VALIDATION 


ASSESSMENT 


Prescribed  conditions  for  which  the 
COMPUTERIZED  MODEL  has  been  tested, 
compared  against  REAiLITY  to  the 
extent  possible,  and  judged  suitable 
for  use  (by  MODEL  VALIDATION,  as 
described  below) 

Demonstrated  agreement  between  the 
COMPUTERIZED  MODEL  and  REALITY  within 
a  stipulated  DOMAIN  OF  APPLICABILITY 
The  process  of  programming  the  formal 
model  on  an  adequate  computer.  It  is 
recommended  to  apply  software 
engineering  methods  such  as  top-down 
design,  structured  programming, 
top-down  Implementation  and  testing, 
etc . 

The  process  of  demonstrating  that  the 
formal  model  has  been  correctly 
implemented  on  the  computer.  This 
includes  source  code  inspections, 
code  walk  throughs  and  tests  of  the 
model  behavior  predicted  on  the  basis 
of  the  theoretical  iriodel  (analytical 
solutions,  behavior  for  small 
signals,  etc.) 

The  process  of  demonstrating  through 
objective  testing  that  the 
theoretical  model  and  its 
implem.entation  form  an  adequate 
representation  of  the  system  to  be 
simulated,  judged  in  view  of  the 
intended  application.  Model 
generated  output  data  are  being 
compared  against  actual  data  obtained 
by  experiments  performed  on  tbe  real 
system 

Tbe  process  of  applying  subjective 
judgment  (i.e.,  expert  opinion)  to 
arrive  at  conclusions  concerning  tbe 
adequacy  of  tbe  missile  design, 
hardware,  simulations 
(hardware-in-tbe-loop  and 
mathematical),  and  testing 


CERTTFTCATION  DOCUMENTATION 


A 


Docuirer.tatton  to  communicatp 
information  concerning  a  model's 
credibility  and  applicability, 
containing,  as  a  minimum,  the 
following  basic  elements: 

(1)  Statement  of  purpose  for  which 
the  model  has  been  built 

(2)  Verbal  and/or  analytical 
description  of  the  CONCEPTUAL  MODEL 
and  COMPUTERIZED  MODET. 

(3)  Specification  of  the  DOMAIN  OF 
applicability  and  RANGE  OF  ACCURACY 
rela.ted  to  the  purpose  for  which  the 
model  is  Intended 

14)  Description  of  tests  used  for 
MODEL  VERIFICATION  AND  MODEL 
VALIDATION  and  a  discussion  of  their 
adequacy 

MODEL  CERTIFICATION  Acceptance  by  the  model  user  of  the 

CERTIFICATION  DOCUMENTATI ON  as 
adequate  evidence  that  the 
COMPUTERIZED  MODEL  can  be  effectively 
utilized  for  a  specific  application 


TTI.  COMFIDF.MCV  1  KVI'I.  TN  MOrin,  BKFAVIOK  (riTMB) 


3,;  INTRODUCTTOK 


CLTMP  (Corfidence  Level?  In  Model  Kehavior)  i?  e  five  le^'el 
hi erarchi rfil  process  for  representing  information  ahont  a  model  such 
that  a  third  party  user  can  readilv  develop  confidence  in  th.p  model's 
behavior.  The  casual  observer  of  the  tcrmat  for  CI.IMB  process  m.av 
conclude  that  this  is  a  special  hut  comprehensive  docuroentatioTi 
procedure.  However,  a  closer  look  by  the  more  than  casual  observer 
interested  in  simulation  m.odcl  development  will  recognize  a  hierarchical 
data  representation  structure  for  achieving  a  specific  purpose.  A 
closer  examination  of  CLIMB  reveals  that  a  type  of  knowledge  base  is 
specified.  This  knowledge  base  serves  as  a  guide  for  producing 
information  and  simulation  model  generated  data  required  to  develop 
confidence  in  the  model's  ability  tc  achieve  the  ’ntended  purpose. 

The  term  "knowledge  base"  as  used  here  includes  two  components: 
facts  about  the  domain  of  Intended  applicaticn  and  heuristics  or  rules 
of  thumb  for  solving  problems  in  the  dom.ain  of  interest.  An  example  of 
the  facts  associated  with  a  particular  model  might  include  the 
differential  ecuatlons  describing  the  dynamics  of  the  modeled  process. 
Also,  textbooks  and  iournals  provide  widelv  shared  facts  generally 
acrepteci  by  simulation  modelers.  Heuristics,  on  the  other  hand,  are 
"educated  guesses"  or  rules  of  pood  practice  acouired  by  the  experienced 
modeler  over  vears  of  experience  in  developing  and  implementing 
simulation  models.  These  educated  guesses  arc  what  the  experienced 
model  developer  uses  as  .a  guide  in  making  many  decisions  during  the 
model  development  process.  Due  to  oversight  and  many  other  reasons, 
seldom,  if  ever  are  the  heuri.stlcs  of  the  modeler  reported  for  a 
parrioular  model  development.  Tiiis  results  in  an  Inadequate  knowledge 
base  for  establishing  model  credibility.  All  experienced  or  expert 
model  developers  have  their  own  "rules  of  rhumb"  for  solving  particular 
simulation  design  and  modeling  problems.  Given  seveiaJ  equal  viable 
choices  chat  may  be  available  to  the  modeler,  his  rules  of  thuiiib  will 
influence  a  particular  choice  in  establishing  the  basic  structure  of  the 
model.  Assumptions  made  in  developing  and  implementing  the  model 
reflect  the  heuristics  cf  the  modeler,  as  well  as  assumptions  made  about 
the  domain  of  intended  application.  Establishing  credibility  with 
second  and  third  party  users  of  a  model  requires  that  more  than  just  the 
tacts  about  a  model  be  reported.  In  addition,  relevant  heuristics  used 
by  the  modeler  must  also  be  captured,  i.e.,  a  knowledge  base  must  be 
available  and  in  a  usable  form, 

A  major  element  in  developing  model  credibility  is  the  assumptions 
made  bv  expert  modelers  related  to  model  verification  and  validation.  A 
missile  model  that  is  validated  by  going  from  an  an.Tlvtical  model  to 
flight  test  data  does  not  have  the  same  level  of  credibility  as  a  model 
that  was  validated  using  intermediate  stages  of  sub-system  testing  and 
model  validation  combined  with  hardware-in-the-1 oop  operation.  This 
assumption  appears  to  be  made  about  the  hardware-in-t he-1 oop  option, 
even  if  the  number  of  real  world  flight  tests  are  less  than  the  flights 
with  the  analytical  model  option.  If  this  is  a  rule  of  thumb  of  a 


nodeler,  this  will  more  than  likely  influenoe  rhe  structure  o'”  the 
pvstem  and  subsystem  models  and  the  data  recorded  from  the  validatior 
tests . 

The  Cl.IMB  process,  as  describea  here,  can  be  viewed  as  a  process 
for  capturing  knowledge  about  a  model  sufficient  to  develop  levels  o'" 
confidence  in  the  behavior  of  the  model  in  question  for  a  particular 
domain  of  application.  In  addition,  ChlMB  functions  as  a  practical 
hierarchical  knowledge  structure  for  identifying  model  ope.'ation  using 
at  least  three  components;  (1)  documentation  format,  (7)  knov;ledge 
base  requirements  and  O)  a  guide  for  simulation  model  development  for 
generating  results  conducive  to  increasing  confidence  in  model 
performance.  Vihen  good  model  development  practices  are  used,  the  CLIMB 
process  captures  that  portion  of  the  knowledge  base  available  during 
model  development  and  validation  efforts.  Using  this  approach,  CLIMB 
does  not  restrict  the  choices  of  the  model  developer,  but  specifies  uh.at 
knowledge  should  be  recorded  once  choices  are  made.  This  includes  all 
stages  of  model  development  and  validation  typically  associated  with 
missile  system  development.  These  stages  Include:  (1)  the  analytical 
mode],  (2)  subsystem  testing  and  model  validation,  13)  hardware-in-the- 
loop  operation  and  (4)  testing  of  the  real  world  system  with  model 
updates  and  validation. 


3..'’  BACKGROUND  ON  THE  DKVEI.OHHENT  OF  CLIMB 

Results  from  a  study  of  24  major  simulation  facilities  in  five 
member  nations  in  the  NATO  community  (Ref  1)  established  the  need  for  a 
procedure  that  would  serve  as  a  guide  for  developing  confidence  in 
simulation  models.  Interviews  with  facility  managers,  model  developers, 
and  simulationist  during  on-site  visits  Identified  at  least  three 
elements  that  must  be  included  in  any  effective  procedure  for  developing 
confidence  in  simulation  model  behavior.  The  elements  are  identified 
as:  (1)  documentation  guide  and  format,  (?)  a  structure  for  representing 
domain  specific  knowledges  and,  (3)  a  guide  for  generating  simulation 
model  data  bases  appropriate  for  building  confidence  in  the  model's 
behavior . 

Documentation  determines  the  life,  death,  duration,  and  quality  of 
a  model's  existence.  Results  from  the  referenced  study  revealed  that 
the  largest  and  most  active  simulation  facilities  had  little  or  no 
guidelines  for  documenting  the  simulation  model  development  and 
validation  efforts.  In  some  instances,  military  specifications  were 
used  to  meet  documentation  requirements.  For  the  most  part,  model 
developers  were  left  to  choose  what  was  to  he  documented  and  how  it  was 
to  be  recorded. 

Reviewing  model  documentation  reports  generated  from  different 
departments  within  the  same  organization  did  not  Instill  confidence  in 
completeness  or  usability  of  the  contents.  The  general  view  of 
documentation  as  derived  from  this  study  can  be  depicted  as  shown  in 
Figure  3.1.  Bits  and  pieces  of  data  are  acquired  during  and  after  the 
model  developmert  effort  and  put  together  as  a  documented  data  base  for 


the  model.  The  data,  freouentlv  with  missing  critical  information,  i.«; 
assembled  through  a  process  that  results  in  a  multi-volume  set  of 
documents.  All  too  frequently,  this  dazzling  arrav  of  paper,  charts, 
and  graphs  lerd  little  or  no  motivation  to  potential  users  to  take  a 
serious  Icok  at  the  model.  A  conclusion  from  this  observation  is  that 
documentation  should  be  an  integral  part  of  the  model  de^'elopmenr 
process . 

Model  validation  techniques  were  not  used  in  any  systematic  fashion 
in  a  vast  maiority  of  the  facilities  surveyed.  In  a  few  facilities,  the 
approach  was  to  attack  the  model  at  all  levels  using  numerous  methods 
and  techniques  to  correlate  the  simulation  results.  Other  validation 
efforts  attempted  to  apply  specific  techniques  ro  particular  parts  of 
the  model.  The  spectrum  of  validation  efforts  can  be  v’iewed  as  using, 
the  random  tool  boy  approach  in  accomplishing  model  validation  as 
depicted  in  Figure  3.2.  Here,  validation  is  mostly  considered  an 
operation  that  takes  place  after  the  model  is  designed  and  developed 
using  whatever  tools  that  are  readily  available. 

Results  from  Interviews  conducted  during  this  study  indicate  that 
in  practice,  a  hierarchy  of  validation  methods  are  used  for  confidence 
building  in  simulation  models.  The  methods  most  frequently  used  are 
O)  expert  opinions,  (2)  data  plots  with  overlays,  and  (31  charts  and 
graphs  with  mathematica j.  analysis  techniques.  The  vast  maiority  of 
responses  to  the  question  "What  procedure  do  you  use  tc  validate  your 
simulation  models"?  was  to  execute  and  change  the  model  until  there  was 
a  "Good  Feeling"  about  tha  model.  How  is  this  "Good  Feeling" 
communicated  to  a  second  or  third  party?  The  typical  response  was  that 
the  interested  party  must  exercise  and  change  the  model  and  compere  data 
with  various  sources,  just  as  the  developer  did  to  uncerstand  and  gain 
insight  Into  the  m.odel  behavior. 

A  second  conclusion  from,  this  study  is  that  expert  opinions,  data 
plots,  and  mathematical  analyses  are  important  methodologies,  but  are 
not  the  main  issues  to  be  addressed  in  developing  confidence  it' 
sir.ularion  models.  These  results  indicate  that  the  basic  issue.s  are  the 
nature  of  the  model  and  real  world  data  bases.  The  model  either 
represents  a  theoretical  system  or  a  real  world  system.  The  real  world 
system  is  associated  with  laboratory  test  data,  hardware-in-the-1 oop 
operation  and  real  world  data.  If  a  theoretical  mode’  produces  data  for 
which  no  real  world  system  exists,  then  onlv  a  limited  confidence  level 
could  be  established  in  the  miodel's  behavior.  However,  if  the  model 
produced  data  that  could  he  compared  with  real  world  systen.s  or  other 
validated  n.odels,  an  Increased  level  of  confidence  in  model  behavior  is 
established.  Additionally,  if  model  generated  data  compares  with 
results  produced  from  a  hardware-in-the-locp  simulation,  then  another 
increase  in  level  oi  confidence  can  be  established.  This  process 
continues  until  m;UJtiple  sets  and  sources  of  real  wcrld  system'  data  are 
compared  with  model  generated  data,  producing  an  ever  higher  level  of 
confidence  in  model  behavior.  The  process  of  relating  confidence  levels 
and  data  bases  is  graphically  depicted  in  Figure  3.3. 
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All  the  validation  methods  and  mathematical  tools  can  be  used  to 
perform  data  analysis  at  each  CLIMB  level;  data  plots,  overlays,  Monte 
Carlo  analysis,  chl-squared  and  Kolmogorov-Smlrnov  test,  hypothesis 
testing,  comparison  of  means  and  variances,  etc.  However,  the 
confidence  level  In  model  behavior  will  not  Increase  beyond  a  level 
determined  by  the  nature  of  the  data  available  to  compare  with  the  mode] 
generated  data.  In  order  to  Increase  credibility,  there  must  be  a 
source  of  data  at  a  progressively  closer  and  closer  level  to  the  real 
world  system.  The  elaborate  use  of  expert  opinions,  data  plots,  and 
mathematical  analvses  onlv  enhance  the  "Cood  Feeling"  about  the  model 
performance  at  a  particular  confidence  level,  but  does  not  provide  a 
basis  for  increasing  the  confidence  level  In  a  model's  ability  to 
reflect  the  real  world.  This  observation  Is  graphically  represented  by 
the  staircase  levels  of  confidence  and  associated  data  base  shown  in 
Figure  3.3. 

3.3  APPLICATIOM  OF  THE  CLIMB  PROCESS 

The  conceptual  structure  of  the  CLIMB  process  was  developed  during 
the  first  meeting  of  the  working  group.  Details  of  the  five  level 
structure  evolved  through  repeated  review  and  application  of  the  concept 
to  existing  simulation  models.  Results  from  individual  modeling  efforts 
were  used  to  develop  the  detail's  at  each  level  of  CLIMB  ^Appendix  A). 
This  provided  both  a  broader  view  of  the  application  and  greater  depth 
in  defining  the  knowledge  base  for  the  hierarchical  levels.  An  existing 
documented  slmtilatlon  model  was  recast  in  the  CLIME  format  throu.gh  level 
three  (Appendix  B) .  This  example  provided  the  opportunity  to  Identify 
significant  operational  consideration  In  the  application  and  use  of  the 
CLIMB  process  with  existing  models. 

The  application  of  CLIMB  to  these  existing  models  detionstrated 
vividly  to  the  members  of  this  working  group  the  effectiveness  of  the 
CLIMB  process,  revealing  in  several  instances  insufficient  cata  in  the 
model  generated  data  base  and  the  lack  of  infornation  regarding  model 
operation.  Conclusions  can  be  drawn  about  the  use  of  CT.TMB  in  two 
areas.  One,  the  most  eificient  and  (-‘fectlve  uses  of  CLIMB  r.re  in  the 
areas  of  establishing  a  basic  framework  for  new  inc'del  development  and 
validation  efforts.  The  knowledge  required  for  desired  confidence 
levels  will  be  available  during  development  with  straight  forward 
documentation  resulting.  I'wo,  investing  the  manpower  and  computer 
resources  for  the  application  of  tT.lMB  to  developed  or  existing 
simulation  models  will  be  most  effective  in  areas  where:  (a)  the  model 
will  be  used  and  may  be  updated  by  third  parties  not  involved  in  the 
development,  e.g.  ,  international  transfer  of  models  and  ^b)  the  model 
will  be  used  over  extended  periods  where  the  developer  would  not  be 
available  to  establish  confidence  in  the  model  behavior,  i.e.,  models  of 
operational  weapons  systems  that  require  modeling  a.id  analysis  support 
from  different  groups  over  the  life  of  the  system. 


3.4  Sl,^J^RY  oy  Tl’F,  C.LTMB  PROCKSS 
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The  ChlMP  (Confi<ience  Level.*;  ■'p  Vodel  Behavior)  process  Is  a 
hierarchical  process  for  developing  confiderco  in  simulation  models 
through  the  integrated  use  of  documentation,  identification  of 
appropriate  knowledge  bases  for  a  given  level  of  confidence,  and  r. 
progressive  use  of  analysis  tools  to  broaden  the  confidence  in  model 
performance.  A  suiratary  of  the  essential  Information  for  each  idertified 
CLIME  level  is  as  follows: 

CLIMB  TiRVF.T,  I:  Model  Summary,  Results  and  Conclusion.  This  level 
includes  information  on  the  objective  of  the  simulation,  model 
developer,  function  of  the  model,  domain  of  application,  major 
assumption  made  in  model  development,  criteria  for  model  validation  and 
the  results  of  model  applicetion.  At  this  level,  only  functional 
diagrams  with  major  subsystems  and  critical  variables  are  identified. 

The  overview  nature  oi  the  information  here  is  intended  to  give  the 
potential  m.odel  user  sufficient  information  to  take  tbe  first  step  in 
reviewing  the  model  capability  without  getting  lost  in  details.  Expert 
opinion  is  typically  the  major  tool  for  confidence  building  at  this 
level  with  descriptive  rather  than  technical  docum.entation . 

CLIMB  I.FVEL  2:  System  Models  and  Submodels  Theoretical  and 
Indirect  Data  Base.  The  data  base  source  is  from  theoretical  models  or 
existing  validated  models.  Method  of  data  comparison  is  included  along 
with  technical  and  descriptive  documentation  of  submodels.  A  benchmark 
scenario  with  model  results  are  given  along  with  verification  proceoures 
of  any  computer  implementation. 

CLIMB  LEVEL  3:  Subsystem  Real  World  Data  Base.  This  level 
includes  real  world  data  for  at  least  one  major  subsystem  to  be  compared 
with  tbe  simulation  model  generated  data.  Documentation  is  proviclec  for 
the  total  complex  model  including  benchmark  results.  Data  collection 
and  validation  methods  are  described. 

CLIf'B  LEVEL  4:  Dardware-Tii-The-Loop  Operation  Data.  This  data 
base  includes  result.s  from  a  Hardware-Tn-The-Loop  (I'.V’TT.)  simulation  with 
ma'^or  subsystem  models  being  replaced  with  actual  hardware  operation. 
Methods  of  data  base  comparisons  are  Identified  along  with  criteria  for 
subsystem  model  validation.  Specifics  on  the  computer  configuration  for 
HWIL  operation  is  also  given. 

CLIMB  LEVEL  5:  Total  Real  L'orld  Systems  Operations.  A  data  base 
is  available  from  the  real  world  system  test  and  operation.  As  a 
minimum,  the  critical  variables  are  compared  with  corresponding  r.vstem 
data.  Results  of  validation  of  system  variables  are  given  along  V7lth 
methods  of  validation  according  to  established  validation  criteria. 

A  detailed  outline  of  the  elements  in  the  total  CLIMB  process  Is 
reported  in  Appendix  A. 
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IV.  COMPUTER  I.ANCUAGEE 

ADA-BASED  SIMULATION  LANGUAGES 

The  fact  that  ADA  has  been  declared  to  be  the  standard  for  embedded 
software  In  the  future  has  some  Impact  on  dynamic  system  simulation 
languages.  The  simulator,  besides  being  used  as  a  tool  for  system 
development,  is  also  used  as  a  system  ifitegration  facility  that  Includes 
the  integration  of  embedded  software.  This  second  domain  of  application 
has  a  growing  importance.  Since  the  operational  software  to  be  embeddec 
has  to  be  written  in  ADA,  there  Is  a  strong  need  tor  the  integration 
facility  software  also  to  be  written  in  ADA  because  there  v/ill  be  no  way 
to  interface  AiDA  to  any  other  language.  Such  interfacing  would  be  in 
direct  contradiction  to  ADA's  design  target  of  software  rellabilltv. 

This  m.eans  that  today's  simulation  languages  will  no  longer  be 
applicable  for  the  purpose  ol  embedded  software  integration.  Tbuf;,  a 
new  generation  of  simulation  languages  will  have  to  be  introduced  even 
though  it  may  not  really  be  possible  to  do  tliis  in  the  given  tlm.e  span. 
One  possibility  to  overcome  the  time  problem  is  to  stay  with  the 
existing  simulation  languages  and  to  modify  the  compiler  to  transform 
from  the  native  simulation  language  code  into  an  ADA  source  code  program 
that  will  operate  in  a  preprogrammed  .ADA  simulation  system.  This  wciild 
be  equivalent  to  the  existliig  languages  that  use  FORTRAN  as  an 
intermediate  language. 

In  order  to  start  as  soon  as  possible  with  new  software,  PAFCAl. 
could  he  used  until  ADA  becomes  available.  Desides  the  aspect  of 
emhedeed  software  integration,  ADA  can  he  helpful  it:  identifying  modern 
software  techniques  that  should  be  Included  in  future  simulation 
languages.  Examples  of  such  broadly  accepted  common  features  would  be: 

-  GOTO — free  structured  programming  techniques 

modularization  of  prograii-s  into  complete, 
self-contained  sub-mcdules 

mandatory  declaration  of  variables,  etc. 

4.2  SIMULATION  SYSTEM  ENVIRONMENT 

There  is  a  great  number  of  problems  related  to  simulation  and 
projects  making  use  of  simulation  tools.  Some  of  these  problems  that 
are  normally  not  mentioned  when  dealing  with  simulation  languages 
Include : 

(1)  Life  cycle  maintenance  costs  for  weapon 
performance  simulation  programs 

(2)  Ease  of  software  maintenance  during  development 
Item  (2)  Is  very  Important  and  needs  to  be  discussed  further. 

During  a  project  involving  a  great  number  of  engineers,  it  must 
never  happen  that  a  copy  of  a  "reference  standard  program"  Is  made  which 


is  not  made  from  a  certified  source  program  and  not  clearly  identified 
and  registered. 

Any  change  to  the  reference  standard  program  has  to  be  immediately 
duplicated  cn  all  existing  copies  of  that  reference  program  so  that, 
during  development,  this  requirement  cannot  be  fulfilled  without 
activating  a  considerable  amount  of  manpower.  Therefore,  one  design 
obiectlve  for  future  simulation  languages  should  he  to  offer  tools  for 
facilitating  such  "software  consistency  management"  work. 

The  software  management  problem  automatically  leads  to  the  problem 
of  describing  the  environment  of  the  program.  It  is  not  sufficient  to 
define  only  the  program  itself  and  the  language  used  for  the 
compilation,  hut  all  environmental  conditions  have  to  be  described  such, 
as  computer  type  and  model,  operating  systems,  libraries,  etc.  If  would 
be  most  desirable  if  future  simulation  languages  be  included  as  an 
Integral  part  of  a  bigger  simulation  system  environment  description  with 
system  features  like  those  mentioned  software  consistency  management 
features  being  Included.  There  are  a  great  number  of  problems  that  are 
most  common  in  today's  simulation  facilities  and  that  could  be  avoided 
by  a  well  designed  environment  and  a  language  supporting  that 
environment.  Therefore,  it  seems  to  be  reasonable  to  suggest  further 
investigations  in  that  area. 

4.3  LANGUAGE  FEATURES 

A  wish  list  of  desirable  language  features  has  been  compiled.  This  list 
is  presented  here  without  comments  just  for  information: 

-  portability 

improved  error  debugging,  error  avoidance,  automatic  detection 
of  errors  (computer/language/compiler  problem) 

-  output/plot  capability,  stripplot,  plot  overlay,  specific  plot 
for  pin-pointing  e.g.,  showing  jumps  of  the  discrete 

control  1 ers . 

-  non-ideal  behavior  of  digital  controllers 

.  synchronization,  time  delay,  time  matching  ("discrete  time 
event"! 

.  fixed  point  arithmetic,  scaling,  overflow,  ADC,  DAC 
simulation  of  noise 

statistical  runs,  Monte  Carlo  runs,  statistical  evaluation, 
connection  to  statistical  programs  , 

connection  to  control  analysis  plus  synthesis  programs  for 
verification,  analysis,  synthesis,  optimization  "ANALYZ" 

-  integration  algorithms  plus  step  sizes 
multiple  integration  algorithms  plus  step 
sizes  interpolation,  extrapolation 


configuration  control 

simulation  of  specific  points  such  as  friction  ("discrete  state 
event") 

real  time  capability  plus  hardware-in-the-loop  capability 
modularity  (i.e.,  easy  decomposition  of  models  into  sub-units) 
structured  programming  techniques 

automatized  "software  consistency  control"  administering  the 
different  releases  and  versions  of  library  programs 

decomposition  of  a  simulation  "program"  into  "model"  and  driv'lng 
environment  "experiment" 

data  base  concept  for  result  data  and  post-processor  programs 


V. 


f'CFTOARF/VERlFICAT10>7VALir)ATI0N/ASSr;sr<MFNT  METHODS 


5.1  IKTRODUCTICK 

Following  mucli  dlscusslon/debate  by  members  of  this  working  group, 
a  general  consensus  was  reached  that  a  detailed  treatment  of  specific 
verification  and  validation  techniques,  their  merits,  drawbacks,  etc., 
as  applied  to  guided  missile  simulation  development  would  far  exceed  the 
scope  and  time  limitations  imposed  on  WC-12  and,  therefore,  not  be 
attempted.  It  was  decided  Instead  to  concentrate  the  group's  primary 
efforts  on  outlining  the  CLIMB  process  previously  discussed.  Although 
the  primary  efforts  of  this  working  group  were  not  directed  at  exploring 
the  merits  of  verification  and  validation  techniques,  for  completeness 
an  attempt  Is  made  In  this  section  to  identify  many  of  the  popular 
approaches  with  a  specific  effort  made  to  identify  references  so  the 
reader  can  explore  in  more  detail  particular  approaches  of  Interest. 

Consistent  with  the  terminology  established  in  Section  II,  Figure 

5.1  Illustrates  graphically  the  interrelationships  between  verification, 
validation,  and  assessment.  As  can  be  seen  in  the  figure,  all  key 
elements  of  verification  stem  from  some  form  of  mathematical  or  com.puter 
code  checks  performed  on  the  simulation  to  verify  that  the  answers 
produced  by  the  simulation  are  consistent  with  theory.  One  key  answer 
is  that  to  the  question,  "Is  the  simulation  code  error  free  and  does  it 
produce  answers  consistent  with  the  math  model?"  It  can  also  be  seen 
from  the  figure  that  a  rather  large  menu  of  verification  options  and 
techniques  are  commonly  used.  Vith  the  increasing  use  of  highly 
structured  and  sometimes  automated  simulation  languages,  techniques  as 
these  are  gradually  being  Incorporated  dlrectlv  into  the  languages,  thus 
making  the  verif Icatlon  process  easier  and  more  straightforward  to 
accomplish.  Each  of  the  verification  techniques  illustrated  in  the 
figure  are  discussed  in  Paragraph  5.2  below. 

Validation,  or  the  other  hand,  requires  real  world  data  to  compare 
against  simulation  resultsj  not  theoretical  predictions  as  Is  the  case 
for  verification.  As  shown  in  Figure  5.1,  some  form  of  hardware  testing 
must  be  conducted  to  obtain  the  necessary  real  world  data.  These  tests 
generally  consist  of  bench  test,  hardware-ln-the-loop  (HIL)  simulation, 
and  free-fllght  trials.  Unlike  verification,  which  as  discussed  above 
is  beginning  to  take  on  a  rather  structured  straightforward  approach  and 
can  very  closely  approach  an  absolute,  validation  is  still  widely 
varying  and  subject  to  individual  preferences  and  technique.  To 
accomplish  model  and  simulation  v’alldatlon  (usally  accomplished 
simultaneously),  some  form  of  objective  mathematical  method  must  be 
applied  to  compare  available  real  world  data  to  simulation  predictions. 
Additionally,  the  real  world  data  base  used  for  validation  must  he 
independent  from  that  used  in  the  original  model  development.  A  good 
example  of  this  is  the  validation  of  a  missile  aefodynamic  model  using 
data  obtained  from  actual  vehicle  flight  testing  when  the  original  model 
had  been  developed  from  wind  tunnel  data  or  aerodynamic  prediction 
computer  programs.  Several  mathematical  methods  being  used  by  members 
of  this  working  group  for  validation  and  others  reported  in  open 
literature  are  discussed  in  Paragraph  5.3  below. 
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The  last  topic  to  be  discussed  In  this  section  of  the  report  is 
assessment.  Assessment,  as  shown  in  Figure  5.!,  plays  a  significant 
role  in  the  missile  overall  design,  development,  and  testing  proces.r. . 

Two  types  of  assessment  arc  identified  in  the  figuie;  one,  in-house  and 
two,  third  party.  Although  each  type  involves  subiective  judgment, 
consistent  with  the  terminology  in  Section  TI,  one  feeds  inforr.intlon 
hacV  into  the  missile  design  process  while  the  other  feeds  infoimatlon 
to  the  customer's  management  decision  process.  The  general  assessment 
problem  along  with  details  of  both  assessment  types  are  discussed 
further  in  Paragraph  5.4  below. 

5..T  VERIFICATION/VAI.IPATTON  METhOPP 

5.2.1  Systematic  Program  Testing:  (A  difficiilt  task.l 

One  of  the  largest  illusions  in  software  development  is  the 
belief  that  it  is  possible  to  get  bv  with  little  program  testing  and 
^^erlflcarion  (Ref  2).  According  to  studies  in  the  T'SA  fRef  .I'l  ,  the 
verification  requires  302  -  502  of  the  total  project  costs  fpig  5.2). 

The  higher  the  portion,  the  more  complex  the  system.  No  methods, 
techniques  and  tools  are  available  to  generate  programs  without  errors. 

It  is  said  that  the  originators  of  programming,  anrong  others,  John 
von  Neumann,  have  been  totally  surprised  by  the  error-rate  of  their 
programs  (Ref  4).  Methods  for  a  mathematical  proof  that  the  program  is 
faultless  have  not  been  fully  developed  up  to  now.  So  the  testing  is 
the  only  means  for  minimizing  the  error-rate  (Ref  5).  The  testing 
begins  with  the  functional  speclficatlcn,  continues  with  the 
verification  of  the  different  modules  and  ends  with  the  integration  test 
(Figs  5.3  and  5.4). 

There  exist  som.e  general  rules,  which  can  help  to  minimize  the 
errors : 


-  VJell  defined  specification  and  program  design,  a  farsighted 
program  structure  and  terminology 

Fareful  coding 

Design  of  the  program  with  regard  to  testing;  Implementation  of 
testing  aids  already  in  coding  (Ref  6) 

Advanced  computer  tools  for  formal  testing  computers,  languages 
and  compilers  with  improved  error  avoidance 

-  Selection  of  adequate  design  and  test  strategies  (Fig  5.5): 

.  "Top-down".  The  modules  are  substituted  by  program  "stubs." 
If  the  overhead  program  has  been  tested,  the  stubs  are 
replaced  by  detailed  modules  (Ref  7) 

•  "Bottom-up".  The  design  and  testing  begins  with  the  lower 
modules.  The  upper  programs  or  calling  programs,  are 
substituted  by  test  drivers  that  are  later  replaced  by  real 
programs  (Ref  8) 


A 
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’’Hardest-First".  The  programming  and  testing  begins  in  the 
middle  with  difficult  modules.  The  program  grows  botti  up  and 
down  (Refs  9  and  10). 

Fach  of  these  strategies  has  its  advantages  and  disadvantages  and 
there  is  re  clear  recommendation  In  the  software  engineering 
literature  as  to  which  of  these  methods  is  the  best. 

There  are  two  approaches  for  the  determination  of  test  conditions; 
the  "black  box"  and  the  "white  box"  approaches.  The  "black  box" 
approach  considers  the  module  to  be  tested  as  black  box  between  the 
input  and  output  without  being  Interested  in  the  inner  details  of  this 
box  or  module  (Ref  11).  The  "white  box"  approach,  on  the  other  hard, 
studies  the  details  of  the  module  or  test  obiect  in  order  to  deriv'e  the 
test  conditions.  Thus,  the  amount  of  test  cases  can  be  minimized. 
However,  the  test  object  has  to  be  understood  very  well  by  the  tester. 

A  tester,  who  is  not  fam.iliar  with  the  details  of  the  program,  should 
begin  with  the  "black  box"  approach  (Ref  12). 

It  is  recommended  that  a  test  laboratory  book  be  kept  in  which  all 
test  conditions  and  results  are  documented.  Fig  5.6  and  5.7  show 
typical  test  systems.  Fig  5.8  shows  the  "dual  program  analysis"  Is  that 
used  at  NASA. 

Management  and  costs  of  testing  and  quality  assurance  (OA).  The  test 
management  has  to  take  into  account:  fa)  costs  of  testing  (Table  5.1), 
(b)  planning  of  testing,  (c)  organization  of  testing,  (d)  specification 
of  testing,  and  (e)  revision  of  testing.  The  testing  of  a  program 
consists  of  the  following  test  phases: 

Static  formal  tests  -  The  program  is  compiled  but  not  yet  run. 

-  Dynamic  formal  tests  -  First  runs  with  debugging  aids. 

Test  of  the  modules  -  Module  verification. 

Integration  test  -  Verification  of  the  overall  system. 

Final  acceptance  test  (Ref  13), 

The  first  two  formal  tests  rely  heavily  on  the  available  computer 
and  its  language.  Typical  method.c  available  in  CYBKR-Fortran  V  and  ACSI. 
are  shown  in  Tables  5.2  and  5.3  and  Appendix  C. 
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.  The  earlier  the  quality  assurance  starts,  the  cheaper  It  Is  In 
proportion  to  development  costs  of  the  program;  l.e., 

-  0.093  If  started  at  program  specification 

-  0.125  If  started  at  debugging 

.  The  more  favorable  the  test  conditions,  the  cheaper  the  quality 
assurance  becomes: 

-  0.125  wltb  favorable  conditions 

-  0.420  with  unfavorable  conditions 

.  The  larger  the  system  (>  64  k) ,  the  better  the  relation  of 
quality  assurance  costs  to  development  costs  become: 

Favorable  conditions: 

32%  for  small  systems  (<  32  K) 

12%  for  large  systems  (>  64  K) 

Unfavorable  conditions 

69%  for  small  systems  (<  32  K) 

42%  for  large  systems  (>  64  K) 

Table  5.1:  Costs  of  Testing  and  Quality  Assurance  Derived  from  IEEE 
Studies. 


Static  formal  tests 

-  source  code  Inspection 

-  inspection  of  the  Fortran-llst  precompiled  by  the  simulation 
language 

-  Inspection  of  the  cross  reference  listing 

-  Inspection  of  the  load  list 

-  inspection  of  the  program  portability  (Compilation  of  the 
program  in  the  "ANSI-mode  (Fortran)  to  ascertain  the 
statements  flagged  as  "non-ANSI"). 

Dynamic  formal  tests 

-  Runs  with  "no  preset  =  zero".  (Some  computers  do  not  have 
this  option  by  which  the  error  rate  Is  Increased) 

Runs  with  "DEBUGGING  AIDS" 


ACSE  DEBUG 


-  Cyber  Trace  back  (Post  mortem  dump,  which  prints  a  readable 
summary  of  the  error  condition  and  the  state  of  the  program 
at  the  time  of  failure  In  terms  of  the  names  used  In  the 
original  program.) 

-  Cyber  Interactive  Debug  facility  (CID). 


Table  5.2:  Computer  Tools  for  Formal  Testing 


Recommendations  of  MBB/UA  for  CRL-Inspectlon  of  Fortran-Proprams 
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The  further  tests  are  called  verification  tests.  A  typical  test 
consists  of  the  following  phases: 

-  Recognize  an  error 

-  Make  a  hypothesis  about  the  location  where  the  error  will 
occur  and  the  source  of  error 

-  check  the  hvpothesls  hy  the  available  computer  tools  and 
detailed  "test-prlnt-outs" 

-  correct  the  error. 

The  first  phase,  recognizing  an  error,  is  the  essential  and  most 
difficult  point  of  the  verification. 

V’e  define  an  error  as  the  difference  between  exact  solutions  and 
simulated  solutions.  The  main  difflcultv  with  simulation  programs  is 
that  the  exact  solution  is  only  known  for  simple  modules  under  certain 
restrictions.  With  complicated  models,  the  exact  solution  mav  onlv  be 
approximated  by  numerical  simulation  techniques.  In  the  next  section, 
methods  are  described  for  verifying  and  checking  these  approximations. 

5.2.2  General  Verification  Methods  for  Missile  Simulations 

There  exists  an  extensive  number  cf  publications  on  the  theme 
"verification  of  software",  but  nothing  specific  has  been  published  with 
respect  to  "verification  of  missile  sim.ulations".  The  only  exception  is 
the  paper  of  S.  Schlesinger  from  the  Aerospace  Corporation,  El  Segunoo, 
t'allfornla  (Ref  14).  With  the  example  of  the  simulation  of  an  analog 
autopilot,  typical  verification  tests  are  described. 

In  the  following,  we  applv  the  verification  methods  to  the 
simulation  of  a  more  modern  missile,  which  uses  microprocessors  instead 
of  the  former  analog  controllers.  It  turns  out  that  simulating  digital 
control  sy.stems,  which  are  also  called  "sampl ed-data  control  systems", 
is  a  great  challenge  from  the  point  of  simulation  and  theory.  The 
following  is  intended  to  remain  not  only  general,  hut  to  illustrate  some 
problems  with  examples  resulting  from,  experiences  with  a  large  b  -  DOF 
simulation  written  in  ACRl  and  Fortran  V.  The  verification  methods  that 
have  been  applied  include; 

Verification  of  the  ecuations  of  the  model 

Hand  check 

Verification  against  existing  and  relevant  theory 
Verification  against  other  simulation  programs 
Degeneracy  tests 

ITien  parameters  are  selected  to  eliminate  the  effect  of  a 
particular  feature  in  a  model,  then  the  resulting  output  from 


tbe  computer  simulation  shall  act  as  if  the  characteristic 
irodelled  by  the  eliminated  feature  is,  in  fact,  totally  absent 

Consistency  test 

fimllar  simulation  cases  yield  essentially  similar 

results  even  though  stated  to  the  computer  model  with  differing 

combinations  ot  descriptive  parameters. 

Verification  of  integration  algorithms,  stepsizes,  sorting  and 
timing.  Sorting  and  timing  errors  may  Introduce  additional  time 
delays  to  the  overall  system 

Logic  tests  -  branch/path  tests 

Integration  tests 

The  purpose  of  Integration  tests  is  to  verify  the  interface 
between  (verified)  modules  from,  a  static  and  dynamic  point  of 
view. 


Stochastic  test 

5.2.2. 1  Implementation  of  resting  aids  already  in  coding 

These  testing  aids  are  recommended: 

Switches  for  degeneracy  tests.  For  example  a  rate  gyro  may  he 
simulated 


eVSV  =  0.  no  errors,  no  time  lag 
=  1.  errors,  no  time  lag 
=  2.  errors,  time  lag 

Switches  for  consistency  tests.  A  typical  application  mav  he 
switches  for  verifying  the  subsystems  "Autopilot  +  Airframe"  in 
pitch  against  the  yaw  channel.  The  test  cerditions  must  he 
such,  that  according  to  the  theory,  the  pitch  and 
yaw  channel  must  yield  identical  results. 

Switches  and  test  driving  signals  for  opening  control  loops.  A 
typical  example  is  that  of  the  most  inner  loop;  it  Is  first 
verified  by  means  of  a  deterministic  signal  (step,  ramp, 
slnusodlal  signal).  After  this  test,  the  next  upper  control 
loop  is  closed. 

Specific  "test-prlnt-outs"  for  logic-,  branch-,  path-  and 
timing-  testing. 

Vie  distinguish: 

Determlnl.stic  verification:  The  data  of  the  plant  are  fixed  and 
the  test  driving  signals  are  deterministic. 


-  Stochastic  veri f i cat  ion :  A  stochastic  sipnal  is  used  as  test 
driver.  Also,  the  plant  data  mav  statistically  varv  according  to 
the  assumed,  tolerances  (3  c  valuesl. 

A  general  rule  says,  that  a  simulation  must  first  be  verified  by  means 
of  deterministic  methods. 

5.2.3  Specific  Verification  Methods  for  Missile  Simulations 

5.2.3.  1  Verification  o'"'  the  equations  of  the  model 

Several  equations  are  needed  to  represent  the  imege  of  a  spatial 
target  n.s  seen  by  the  missile  seeker  head.  Kor  verifying  these 
equations,  one  must  know  the  terminology  and  derivation  of  the 
equations,  which  should  be  documented  and  sc  detailea  that  they  can  he 
verified  bv  another  person. 

5. 2. 3. 2  Hand  checks 
Examples  are: 

Verifications  of  the  correct  implementation  of  the  aerodynamics. 

-  Verifications  of  the  integration  algorithms  by  a  sinusodial 
input  sig:nal  which  is  integrated  twice.  The  result  should 
again  be  a  sinusodial  signal. 

Graphical  ''eri f ication  of  the  image  processing  of  the  seeker 
seeing  a  spatial  target. 

5. 2. 3. 3  Verification  against  existing  and  relevant  theory 

Guidance  and  control  theory  has  become  a  well  established  topic 
with  a  history  of  roughly  50  years  and  an  extensive  number  of 
puhl icatlous.  However,  there  exists  a  large  gap  between  the  falrlv 
sophisticated  theorv  and  the  simple  PID-control lets  and  PN-navigat ion , 
which  are  predominant Iv  used  in  practice.  One  of  the  reasons  for  this 
gap  is  the  fact  that  modern  control  theory  is  quite  difficult  to  apply. 
The  theory  of  sampled  data  control  systems  is  especially  much  more 
difficult  to  apply  than  the  conventional  theory  for  continuous  systems. 
Fig  5.9  illustrates  this.  Calculating  the  exact  time  response,  x,  to 
the  deterministic  inputs,  x  ,  which  may  be  a  step,  a  ramp  or  a 
sinusodial  signal,  requires  considerable  skill  in  z-transform  analysis 
fPef  15)  and  a  substantial  number  of  sophisticated  numerical 
calculations  even  though  only  a  simple  digital  PD-control ler  has  to  be 
analyzed.  Also,  random  inputs,  such  as  the  measurement  noise  in  Fig  5.9 
may  be  taken  into  account.  For  example,  there  exists  a  theory  for 
calculating  the  variance  of  the  output,  x,  due  to  the  noise  output, 
n  (t) . 


Special  computer  programs  for  control  analysis  and  design  exist. 

An  example  is  the  Computer  Aided  Control  System  Design  (CADSD)  from  ETH 
Zentrum,  Institute  for  Automatic  Control,  Zurich,  Switzerland  (Fef  16) 
which  consists  of: 


SIMNON  -  A  simulation  program  for  a  continuous  system  wltli 
discrete-time  regulators  (sampled  data  systems) 

IDPACK  -  A  program  system  for  data  analysis  and  identification 
of  linear  deterministic  stochastic  multi-input,  single-output 
systems 

SYNPAC  -  A  state-space  oriented  control  systems  design  program 
PPPAC  -  A  frequency-domain  oriented  design  program,  and 

-  MODPAC  -  A  program  for  transformations  between  different  control 
system  representations 

Another  method  exists  for  analylng  the  digital  control  by  means  of 
approximations.  This  method  has  the  benefit  that  a  special  control 
analysis  program  is  not  necessary  and  hand  checks  may  he  used,  thus 
providing  more  Insight  into  potential  problems.  For  example,  the 
digital  controller  (Fig  5.9)  can  be  converted  into  an  analog  controller 
by  use  cf  a  bilinear  transformation  and  the  sampler  +  zero  order  hold 
may  he  approximated  by  a  first  order  Pade  time  lag  (t  =  T/2)  or  a  time 
delay (e  '  T/2).  Thus  the  conventional  methods  in  the  s-plane  may  be 
used.  Blakelock  (Ref  17)  calls  the  first  approximation  "digitization" 
method  and  compares  it  with  the  exact  z-transform  method. 

There  are  many  other  theories  available,  which  may  be  used  for 
verification  of  simulations: 

Nonliear  Control  Theory 

Kalman  Filter  Theory 

Guidance  Theory 

Conventional  and  modern  guidance  theory  with  deterministic  and 
stochastic  Inputs.  A  typical  example  Is  to  verify  the  miss 
distance  due  to  noise  of  the  seeker. 

Strapdown  Algorithms  Theory 

5. 7.3. A  Verification  against  other  simulation  programs 

Examples  of  this  method  of  verification  Include; 

6  DOF  against  3  DOF 

3  DOF  against  3D  (3-dlmenslonal  with  trimmed  aerodynamics) 

6  DOF  (CSMP)  against  6  DOF  (ACSL)  (Verification  of  the 
conversion  from  one  simulation  language/ccmputer  to  another 
simulation  language/coraputer . ) 

5. 2. 3. 5  Degeneracy  test 

Degeneracy  implies  simplifying  the  phenomena  being  modeled  by 
selecting  certain  special  parameter  values.  An  example  follows: 


For  the  stabilization  of  the  seeker  platform,  the  following 
degeneracy  test  has  been  conducted; 


Simulate  the  gyros  and  look-angle  pick  offs  without  errors  and 
time  lag 

Let  the  two-axis  stabll ization  degenerate  to  a  pltch-only- 
stabllization. 

Under  these  simplified  conditions,  the  stabilization  should  be  perfect. 
In  this  manner,  errors  in  the  program  can  be  detected  much  easier. 

5.?. 3.6  Consistency  tests 

See  Paragraph  5.2.2.  1  under  the  testing  aid  titled  "Switches  for 
consistency  tests." 

5.2.3. 7  Verification  of  integration  algorithms  and  stepslzes,  sorting, 
timing,  and  repeatability 

Similar  to  guidance  and  control  theory,  the  theory  about 
integration  algorithms  has  become  a  well  established  topic  with  a  long 
history.  However,  there  exists  a  large  gap  of  knowledge  between  the 
numerical  mathematician  and  the  practical  simulatlonist .  Furthermore, 
it  turns  out  that  algorithms  that  had  been  extremely  powerful  for  the 
simulation  of  continuous  systems,  such  as  the  Adams  multistep  methods, 
are  no  longer  optimal  for  problems  having  frequent  discontinuities  (Ref 
18).  Modern  algorithms,  which  are  commonly  called  Adams  PF.CR  (predict, 
Evaluate,  Correct,  Evaluate)  automatically  vary  the  stepslze  and  order 
for  solving  a  problem  to  a  given  requested  accuracy.  W.  Bub  shows  in 
Ref  19  that  it  is  possible  to  evaluate  the  numerical  stability  and  other 
features  of  Integration  algorithms  by  means  of  the  z-transform  *  and 
recommends  Adams  multistep  methods  for  the  solution  of  linear  transfer 
functions.  The  other  class  of  integration  algorithms  are  called 
Runge-Kutta  methods.  They  are  most  widely  used  for  the  numerical 
solution  of  first-order,  first-degree  equations.  In  contrast  to  the 
Adams  multistep  algorithms,  they  are  self-starting. 

"Runge-Kur ta"  methods  are  easv  to  implement  on  a  digital  computer 
and  are  probablv  preferable  to  predictor-corrector  techniques  for  most 
purposes.  Their  main  disadvantage  is  that  it  is  difficult  to  keep  a 
check  on  the  truncation  error.  The  simplest  way  to  check  a  solution  is 
to  repeat  it  with  a  halved  step  length  -  though  more  efficient  means  are 
suggested  in  specialist  texts.  A  further  point  is  that  the  widely  used 
fourth-order  Runge-Kutta  m.ethod  requires  four  evaluations  per  step  of 
the  funct_ion.."f".  ,  compared  with  at  most  two  per  step  of  a 
predictor-corrector  solution  of  comparable  accuracy.  The  latter  method 
may,  therefore,  be  more  suitable  in  cases  where  it  is  very  complicated, 
so  that  its  evaluation  is  expensive  in  computing  time.  (Ref  20) 


*  Numerical  mathematicians  use  other  methods. 


After  this  general  literature  overview,  we  shall  deal  vjlth  the 
probletr,  of  simulating  a  fast  digitally  controlled  missile,  whose 
smallest  sampling  time,  T,  must  be  very  small  due  to  the  required  high 
bandwidth  of  seeker  head  stabilization  and  autopilot.  The  main  Issues 
are : 

(a)  Optimal  Integration  algorithm  and  stepslze  (assumption:  the 
program  uses  a  unique  Integration  algorithm  and  stepslze) 

(b)  Verification  of  the  sorting  and  timing 

(c)  Simulation  of  the  non-ideal  behavior  of  the  digital 
controllers 

(d)  Multiple  Integration  algorithms  and  stepsizes  to  save 
execution  time. 


(a)  Choice  and  verification  of  the  Integration  algorithms  and 
stepsizes 

A  general  rule  of  thumb  is  that  the  stepslze,  h,  should  be  at  least 
h  =  1/?  T  .  ,  with  T  .  =  minimum  of  the  smallest  sampling  Interval  T, 

smallest  ?lme  con  stan^^T ,  smallest  =  1 /w  ,  smallest  =  1/m  .  The 
last  2  expressions  refer  to  the  bandwidth  o?  the  noise,  respectively  the 
frequency  of  the  slnusodlal  signal  (see  Fig  5.9).  Tn  this  case,  the 
sampling  time  was  the  main  driver  for  the  stepslze.  The  cheapest 
algorithm  Is  the  Adams-Bashforth  1  (AB  1),  which  requires  only  one 
evaluation  per  step  of  the  derivative  function  "f"  (see  below).  It  has 
been  compared  with  the  Runge-Kutta  2,  which  requires  two  evaluations  per 
step.  The  starting  algorithm,  of  the  AB  1  has  been  executed  only  at  the 
beginning  of  simulation  and  not  as  theoretically  required  at  each 
discontinuity  (l.e.,  sampling  Interval).  Experience  with  a  large 
program  (30  Integrators)  was  that  h  =  0.5  T  was  sufficient  for  RK  2,  but 
not  for  AB  1,  which  requires  in  any  case  h  =  0.25  T.  With  this 
stepslze,  the  AB  1  Is  still  slightly  inferior  to  the  RK  2  with 
h  =  0.5  T. 

Adams-Bashforth  (AB  1) 

t  =  0  X  -t  X 

o  o 

t=h  X,  =  x  +hx  -^x, 

lOOl 

Starting  algorithm 

X,  =  x^  4  h/2  (x^  >  xj)  -  Xj 

t  =  2h  x„  =  X,  +  h/2  (3  X  -  x.) 

2  1  o  1 

t  =  3h  x^  =  +  h/?  (3x^_^  - 

h  (32  -  1) 

This  can  be  expressed  as  z-transfer  function  x(z)/x(z)  =  - 

?z  (z  -  1) 


Runge-Kutta  2.  order  (RK  2) 


X  =  X  +  a  k  +  (1  -  a)  k, 
n+1  no  1 

with  k  =hx,  X  =x(t=t,y=x) 
o  n  r  n  n 

k,  =  h  X  (t  =  t  +  h/2  (1  -  a),  x  =  x  +  hx  1/2  (1  -  a) 


n 

n  +  2/3 

n  +  1 

X 

.n 

Tn  +  2/3 

rn  + 

X 

n 

\  +  2/3 
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The  chosen  way  of  verifying  the  numerical  solution  was  to  repeat 
runs  with  a  halved  step  length  and  to  introduce  a  severe  test  signal 
such  as  a  slnusodial  perturbance.  At  the  beginning,  the  difference  was 
considerable.  One  could  think  that  this  was  the  proof  that  the  smaller 
stepslze  had  to  be  taken.  However,  it  turned  out,  that  also  sorting  and 
timing  errors  contributed  to  the  difference. 

(b)  Verification  of  the  sorting,  timing  and  sample  devices 

With  simulation  languages,  there  exists  an  automatic  program 
sorting.  ACSL  automatically  sorts  the  model  definition  code  that  is 
placed  in  the  DERIVATIVE  section.  The  statements  within  PROCEDURAL 
blocks  are  not  sorted.  The  PROCEDURAL  block  is  positioned  according  to 
the  input  and  output  variables  of  the  PROCEDURAL  header.  Sorting  errors 
result  especially  in  that  case  where  not  all  output  and  input  variables 
are  in  the  header.  On  the  other  hand,  too  many  variables  in  the  header 
may  result  in  "program  unsortable."  Another  problem  is  the  simulation 
of  digital  algorithms.  A  large  missile  simulation  contains  multiple 
sampling  devices  and  different  time  delays. 

The  following  verification  methods  have  been  used: 

-  Runs  with  "no  preset  =  zero"  to  detect  initialization 
errors . 

-  Verification  of  the  repeatability;  2  runs,' which  are  repeated 
in  one  JOB,  must  yield  identical  results. 

-  Runs  with  additional  evaluation  of  the  DERIVATIVE  section  at 
each  communication  interval  (which  is  identical  to  the 
integration  Interval).  The  comparison  with  runs  where  this 
additional  evaluation  is  omitted  must  yield  identical  results. 


detailed  "test  print  outs"  for  the  testing  of  the  different 
sampling  and  delay  devices. 

(c)  Verification  of  the  simulation  of  the  non-ideal  behavior  of 
the  digital  controllers 

The  non-ideal  behavior  consists  of  computation  delays, 
asynchronous  sampling  and  delays,  errors  due  to  fixed  point  arithmetic 
and  overflow.  Usually,  these  effects  are  assumed  to  be  negligible. 
However,  in  applications  with  fast  control  systems,  these  effects  may 
cause  limit  cycles,  time  matching  problems  and  transient  effects  and 
therefore  cannot  be  neglected  CRef  ?1).  Modern  simulation  languages 
are  Improved  with  respect  to  these  features.  The  ACSL  version  8A  now 
offers  the  macro  SKF.DTE  (Schedule  Time  Event),  which  can  be  used  to 
simulate  a  computation  delay,  AT  ,  after  the  Anal og-Tiigltal-Converslon 
(ADC).  Such  a  feature  of  the  simulation  language  must  be  verified  hv 
means  of  detailed  test  print  outs. 

(d)  Verification  of  multiple  integration  algorithms  and  stepslzes 

The  simulation  of  fast  digital  control  systems  requires  small 
variable  stepsises  and  special  integration  algorithms.  The  slower 
motion  of  the  missile  may  be  simulated  with  larger  stepsizes  and 
multistep  ADAMS  algorithms.  I'lth  ACSL,  more  than  one  DFRIVATIVF  section 
may  be  used,  each  with  its  own  independent  integration  algorithm  and 
stepslze.  Although  this  technique  can  save  execution  time  when 
correctly  used,  any  implementation  must  be  approached  with  caution 
since,  in  general.  Incorrect  answers  will  be  obtained  unless  the  model 
is  divided  into  blocks  with  a  full  understanding  of  the  effects  of 
computation  delays  for  variables  that  cross  block  boundaries  (Ref  2?). 

Languages  are  offered,  which  perform  the  necessary  syntax  analysis 
and  partition  the  problem  into  a  sequential  and  a  parallel  part, 

Fpeclal  methods  must  be  developed  tc  verify  these  sophisticated 
simulations. 

5. 2. 3. 8  Logic  tests  -  Pranch/Patch  tests 

In  designing  an  appropriate  structure  and  logic  of  the  overall 
program,  the  "top-down"  design  is  recommended.  The  more  sophisticated 
modules  are  substituted  by  program  stubs.  Special  logic  drivers  ana 
logic  verification  tests  may  be  used. 

5. 2.3.9  Integration  tests 

The  verification  of  the  overall  missile  system  must  be  based  on 
verification  tests  generated  by  means  of  the  subsystem  simulations. 
Special  testing  aids  (switches,  test  drivers  and  test-prlnt-outs)  are 
helpful  In  comparing  the  overall  program  against  the  subsystem  program. 

5.2.3.10  Stochastic  verification 

The  aim  of  stochastic  verification  is  to  verify: 

the  statistical  behavior  of  the  input 
nolse(varlance,  bandwidth) 


t 


the  statistical  behavior  of  the  output  noise 
against  the  theory  (if  possible)  or  other  programs 

-  whether  the  number  of  statistical  runs  is 
sufficient 

The  following  two  examples  illustrate  this  verification: 

First,  Fig  5.10  shows  the  "measured"  mean  squared  value  as  a 
function  of  the  number  of  runs  N^.  The  theoretical  value  is  100  m.  It 
turns  out,  that  =  100  yields  errors  up  to  10?!  and  is  to  small. 

Finally,  Fig  5.11  shows  the  noise  reduction  of  a  typical 
Kalman-Fllter  with  time-dependent  gain.  If  1000  runs  are  evaluated,  the 
simulation  results  agree  well  with  the  theory. 

What  we  can  learn  from  these  examples,  is  that  a  statistical 
verification  is  extremely  expensive.  On  the  other  hand,  if  only  a  few 
statistical  runs  are  performed,  the  statistical  errors  may  be 
considerable. 


5.3  VALIDATION  METHODS 


5.3.1  The  General  Validation  Problem 

Validation  represents,  perhaps,  the  biggest  challenge  facing 
today's  slmulationlst.  Reviewing  the  definition  on  validation  presented 
in  Section  11,  one  can  see  the  difficulty  of  the  task.  Validation  tests 
the  agreement  between  the  model  and  the  real  world  system  being  modeled. 
The  tw'O  key  words  in  this  definition  are  tests  and  real  world.  The 
requirement  to  apply  mathematical  tests  to  compare  the  model  and  real 
world  separates  validation  from  assessment.  This  also  makes  validation 
an  obiectlve  process,  separating  it  from  assessment  -  a  subjective 
process.  TVie  definition  of  validation  by  Implication  requires  the 
existence  of  a  real  world  data  base  before  validation  comparisons  are 
possible.  This  real  world  data  base  is  generally  constructed  through 
four  primary  methods;  (1)  Bench  tests;  (2)  HIL  tests;  (3)  Field 
measurements  (target  signatures,  etc.);  and  (4)  Flight  trials  (captive 
and  free-f 1 Ight) .  It  should  also  be  noted  that  for  model  validation  to 
be  possible,  the  real  world  data  base  used  for  comparison  must  be 
independent  from  that  used  In  the  original  model  development  process. 
During  early  stages  of  missile  system  development,  model  validation  and 
system  testing  for  assessment  purposes  become  inseparable.  Flight 
tests,  once  exclusively  reserved  for  system  performance  demonstration, 
are  now  often  aimed  at  providing  real  world  data  for  m.odel /simulation 
validation.  Even  with  the  Increasing  emphasis  on  the  availability  of 
independent  data  sources  for  validation,  the  process  Is  not  absolute. 
Simulation  predicted  system  performance  will  never  exactly  match  actual 
system  performance  under  all  conditions.  Results  being  reported  In 
current  literature  on  missile  simulation  validation  suggests  that  the 
proper  Question  Is  not  "Is  the  model  valid"?  but  "How  valid  is  the 
model"?  Some  measure  of  acceptable  error  between  the  model  and  real 
world  must  be  established.  It  Is  precisely  this  problem  that  gives  rise 
to  numerous  mathematical  techniques  to  quantify  the  acceptable  error  for 
validation  purposes.  Several  of  the  most  popular  validation  techniques 
used  In  guided  missile  simulations  are  discussed  below. 

5.3.2  Specific  Validation  Methods 

5.3.2. 1  Pilot  (Vverlays  and  Graphical  Comparisons 

Plot  overlay  and  graphic  techniques  involve  the  comparison  of 
real  world  data  to  that  produced  by  the  simulation  using  plots.  It  Is 
by  far  the  most  popular  validation  technique.  Graphical  comparisons  and 
cross  plotting  of  variables  Is  limited  only  by  the  Imagination  of  the 
slmulationlst  and  the  availability  of  real  world  data.  The  major 
difficulty  with  the  technique  Is  "How  close  is  close  enough"? 

Generally,  "expert  opinion"  must  decide  this  issue.  Common  practice 
using  Overlay  and  Graphical  validation  techniques  is  to  plot  real  world 
test  data  and  establish  a  somewhat  arbitrary,  but  small  allowable  error 
that  the  simulation  must  stay  within  to  be  considered  valid.  Expert 
judgment  Is  usually  the  basis  for  establishing  the  allowable  error. 
Acceptable  error  is  generally  established  between  ore  and  ten  percent 
depending  on  the  particular  state  variables  being  considered  and  the 
Intended  application  of  the  simulation.  The  major  advantage  of  this 


technique,  aside  iron  its  simplicity,  is  the  at>tomatic  and  very  graphic 
validation  audit  trail  develcped  for  the  simul at  ion /model  docunienf a t i on 
package.  For  additional  information  and  examples  of  this  technique, 
consult  (Kef  . 

5. 3. 2. 2  Correlation  Coefficients 

'Ihe  generation  of  correlation  coefficients  is  a  well  knowr 
mathematical  technique  to  measure  the  time  correlation  of  two  processes. 
If,  for  example,  the  time  history  of  a  missile  system  state  variable  or 
internal  subsystem  variable  were  recorded  during  a  live  flight  test  and 
the  time  history  of  the  same  variable  genejated  by  the  simulation  were 
processed  to  obtain  a  favorable  correlation  coefficient  (approaching 
1.0),  the  two  processes  (one  real,  the  otlier  simulated)  are  considered 
equal.  Only  one  computer  run  and  free-flight  trial  is  neces.sary  to 
conduct  this  test.  Generally  speaking,  correlation  coetficients  rae^'sure 
the  degree  to  which  two  time-varying  signals  compare.  A  periect  match 
with  flight  test  data  obviously  represents  a  valid  computation  of  a 
variable.  The  application  of  correlation  mathematics  represents  a 
strenuous  test  for  simulation  validity.  Not  only  must  amplitude 
cl’.aracterl.'^tics  matcli  to  obtain  a  pood  correlaticn  coefficient,  but 
phase  is  also  extremely  important.  Verv  small  deviations  in  missile 
system  m.odels  will  create  significant  phase  changes  on  some  state 
variables,  especially  for  ron-linear  models,  resulting  in  rather 
dramatic  shifts  in  correlation  coefficients.  For  this  reason, 
correlation  coeflicient  techniques  are  not  often  used  for  validation.  A 
detailed  discussion  of  correlation  mathematics  and  examples  of  their  use 
mav  be  found  in  (Refs  23,  24  and  25'. 

5. 3. 2. 3  Thell's  Inequalltv  Coefficient 

A  technique  developed  bv  The! 1  bas  been  used  by  economists  to 
validate  simulations  that  Include  econometric  models.  Tbeil's 
inequality  coefficient,  "U",  provides  an  Index  that  measures  the  degree 
to  which  a  simulation  model  provides  retrospective  predictions  of 
observed  historical  data.  "U"  varies  between  zero  and  one:  if  11=0,  the 
predictions  are  perfect,  if  U=l,  the  predictions  are  very  bad.  Althougl- 
Thell's  theory  was  developed  for  economic  models,  in  recent  years  much 
success  has  been  demonstrated  in  its  application  to  dynamic  scientific 
models.  Examples  of  Its  use  for  missile  simulation  validation  are 
contained  In  (Refs  26,  27  and  28). 

5. 3. 2. 4  Chi-Square  and  Kologorov-Smirov  Tests 

The  chi-square  and  Kologorov-Smirov  tests  are  two  special  types  of 
hypothesis  tests  often  used  to  establish  the  equivalence  of  a 
probability  density  function  of  sampled  data  (in  thie  case,  simulation 
output)  to  some  theoretical  density  function  (In  this  case,  real  world 
data).  Each  of  these  tests  derive  a  f igure-of-merit  to  characterize  the 
goodness-of-f i t  between  two  probability  density  functions.  The 
chi-square  test  general  procedure  Involves  the  use  of  a  statistic  with 
an  appropriate  chi-square  distribution  as  a  measure  of  the  discrepancy 
between  an  observed  probability  density  function  and  the  theoretical 
density  function.  A  hypothesis  of  equivalence  is  then  tested  by 


studying  rbe  distribution  of  this:  statistic.  The  main  problem  of  the 
chi-sauare  test  is  that  it  is  relatively  sersitive  to  nop-norma 1 i t v . 

The  Kologorov-Smirov  test,  on  the  other  hand,  is  a  c i str ibu t 1 or-f ree 
fnonparametric)  test.  It  involves  specifying  the  cumulative  frequencv 
distribution  of  the  simulated  and  actual  data.  Unlike  the  cbi-square 
test,  the  figure  of  merit  or  goodness-of-f 1 t  is  not  a  statistical 
variable  and  Is,  therefore,  not  sensitive  to  normality.  For  more  detail 
concerning  exact  implementation  procedures  for  each  of  these  tests  see 
Refs  ;’3,  25  and  29. 

5. 3. 2. 5  Monte  Carlo  liourdarv  Generation 

Monte  Carlo  Boundary  Generation  is  a  well  known  technioue  involving 
multiple  runs  of  a  simulation,  including  ail  known  noise  and  error 
sources,  to  establish  accumulated  statistical  properties  of  selected 
state  variables  as  a  function  of  tine.  Using  the  overlav  graphics 
methods  discussed  in  Paragraph  5.3.2.  I,  the  mean  and  standard  deviation 
of  the  selected  variables  are  plotted  as  a  function  of  elapsed  simulated 
missile  flight  time.  i real  world  flight  trial  data  for  slnilar  launch 
conditions  overlays  the  Monte  tiarlo  generated  simulation  data  within  the 
established  bounds  (usuallv  one  sign;.),  the  simulation  is  considered 
valid  for  that  variable.  Generating  similar  plots  for  al’  critical 
state  variables  and  internal  svstem  variables  vn  1 1  fullv  validate  the 
simulation.  Two  maior  differences  distinguish  the  Monte  Ciir''o  Boundary 
Technique  from  the  Plot  Overlav  and  Graphical  Comparison  Technique  as 
described  in  this  report.  One,  the  validation  boundaries  are  applied  tc 
the  simulated  data  fot  the  Monte  Carlo  technique  with  flight  test  data 
overlayed  within  the  defined  boundaries  to  establish  simtilation 
validity,  and  two,  the  validation  boundaries  represent  statistically 
generated  error  properties  based  on  multiple  simulation  runs  instead  of 
a  single  flight  test  trial  with  small,  but  sc'mowhat  arbitrary  error 
boundaries  applied  as  is  ttie  case  for  the  Plot  Overlav  and  Graphical 
Validation  Method.  The  maior  disadvantage  of  the  Monte  Carlo  Validation 
Technique  should  be  rather  obvious  bv  now;  that  is,  computer  costs. 
Uurdreds  of  runs  are  sometincs  required  to  obtain  accurate  statistical 
properties  and  to  assure  adequate  confidence  levels  are  obtained.  The 
application  of  speciai  programs  for  statistical  analysis  is  also 
sometimes  required. 

5. 3. 2. 6  Spectral  Analysis 

Spectral  analysis  is  a  class  of  mathematical  processes  that 
consider  the  spectral  content  of  data.  These  include  techniques  as 
Power  Spectral  Density,  Gross  Spectral  Density,  and  others.  Spectral 
analysis  provides  a  ir.eans  of  obiectlvelv  coiriparing  time  series  data 
generated  by  a  computer  simulation  with  an  observed  time  series  obtained 
from  real  world  data  collection.  Spectral  analy.sis  is  aimed  at  the 
quantification  and  evaluation  of  uncorrclated  data  after  the  data  has 
been  transformed  into  the  frequency  domain.  Bv  comparing  the  computed 
spectra  of  simulation  output  data  and  corresponding  real  world  data,  it 
can  be  Inferred  how  well  the  simulation  resembles  the  system,  or 
subsystem  it  is  intended  to  emulate.  Unlike  manv  of  the  other 
validation  techniques  discussed  in  this  report,  spectral  analysis  does 


not  inherently  produce  f igurep-of-meri t  to  quantify  goodneps-of-f it 
between  simulation  output  and  real  world  data.  Spectral  analysis  is 
extremelv  dependent  on  expert  lodgment  to  answer  the  question  "How  close 
is  close  enough"?  For  more  discussion  covering  the  many  spectral 
analysis  techniques  and  examples  of  their  use,  consult  Refs  30,  31,  3? 
and  33. 

5.4  ASSFSSMFMT: 

5.4.1  The  Teneral  Assessment  Prohlem 

Assessment,  as  defined  and  used  in  this  report,  includes  all 
activities  involving  the  application  of  suh’ective  ludgment  (i.e., 
expert  opinion)  to  answer  the  question  "l.'ill  the  systeta/subsy stem  design 
meet  spec  1 f ica t i ons"?  Assessment  involves  subfective  evaluations  of  all 
aspects  o'"  the  weapon  system  development  process,  including,  but  not 
limited  to:  system  simulations,  hardware  bench  tests,  captive  flight 
tests,  free-f light  tests,  hardware-in-the-loop  tests,  etc.  Many  of  the 
mathematical  processes  and  techniques  applied  to  missile  simulation 
validation  are  useful  in  acquiring  data  to  help  .'>nswor  the  assessment 
question.  The  key  to  understanding  the  difference  between  validation  and 
assessment  is  in  the  use  of  the  data.  That  is,  validation  concentrates 
t'n  the  performance  of  the  simulation  fl.e..  Pees  the  simulation  properly 
emulate  the  design?),  while  assessment  concentrates  on  the  performance 
of  the  system  begin  simulated  fi.e..  Does  the  design  meet  specifica¬ 
tions?)  Assessment  begins  verv  earlv  in  the  weapons  development  process 
and  continues  throughout  the  ’ife  of  the  project.  As  illustrated  in 
Figure  5.1,  there  are  two  distinct  tvpes  of  assessment.  One,  in-house 
accomplished  by  the  svstem  prime  contractor,  aiiO  the  other,  third  partv 
accomplished  by  the  customer  or  a  third  partv  contractor  emplcved  bv  the  • 
customer. 

Tn-house  assessment  is  identified  in  Figure  5.1  by  the  boxes 
representing  four  major  data  sources;  the  mathematical  simulation,  the 
HIL  simulation,  missile  design  data,  and  missile  hardware  data.  The 
figure  clearly  illustrates  that  in-house  assessment  plays  a  key  role  in 
the  system/subsystem  design  process.  Although  subjective  in  nature, 
in-house  assessment  offers  the  first  opportunity  to  compare  simulation 
predicted  performance  (general Iv  generated  during  the  simulation 
verification  and  validation  process)  to  customer  requirements  (system 
design  specifications)  and  feeds  back  information  to  the  design  process. 
This  feedback  often  results  In  system  design  modifications  which  in  turn 
result  in  simulation  modi f ications  creating  the  need  for  additional 
passes  through  the  simulation  verification  and  validation  process. 

During  system/subsystem  design  activities,  in-house  assessment  and 
simulation  verification  and  validation  are  Inseparable,  each  feeding  the 
other  until  a  design  is  finalized.  < 

Third  party  assessment,  on  the  other  hand,  involves  independent 
evaluation  of  the  entire  missile  svstem/suhsystem  development  and 
demonstration  process,  and  as  such,  has  little  direct  impact  on  the 
system  design.  Some  indirect  influence  is  present,  however,  due  to  the 
use  of  third  party  assessment  data  for  customer  management  decisions  and 
the  impact  these  decisions  may  have  on  system  speci f icat  ions.  As 


pointed  out  by  Richard  Rlchels  of  the  Electric  Power  Research  Institute, 
"Decision  makers  are  becoming  Increasingly  annoyed  that  different 
analyses  get  quite  different  answers  to  the  same  problem."  When  this 
happens.  It  Is  natural  to  want  to  take  a  closer  look  at  the  simulatiou 
models  employed  and  find  out  why  such  differences  result.  This  all  to 
familiar  situation  has  led  to  the  Increasing  use  of  third  party 
assessment  to  provide  an  independent  check  and  balance  on  the  weapons 
development  process  and  to  provide  customer  management  (l.e.,  decision 
makers)  with  unbiased  nonparochlal  information  on  system/subsystem 
designs  and  performance  limitations. 


VI  SUItllAFY  AND  RECOMMENDATIONS 


!4 


In  summary,  the  Working  Croup  found  that  the  conclusion  reached  in 
ArAPD-AG-279 ,  "Survey  of  Missile  Simulation  and  Flight  Mechanics 
lacilltles  in  MATO"  regarding  the  general  lack  of  uniform  accepted 
missile  simulation  validations  methods  was  indeed  correct.  In  addition, 
the  need  for  such  methods  is  becoming  increasingly  apparent  in  all  of 
the  participating  Working  Group  countries.  This  was  clearly  evident  in 
a  very  recent  article  in  AGARD  Highlights,  "Missile  System  Simulation 
and  Validation,"  by  Dlpl.-Ing  Roland  Cauggel  of  BGT ,  iiarch  1984.  lit  is 
interesting  to  note  that  this  paper  discusses  "Simulation  Levels"  very 
similar  to  the  "CLIMB  Levels"  in  Section  111,  apparently  conceived 
independently  from  the  Working  Group  activity.) 

It  is  one  thing  to  establish  that  uniform  validation  methods  are 
reeded;  yet  another  thing  to  develop  these  methods.  As  stated  in 
Section  I  Introduction,  the  Working  Group  found  the  latter  a  formidable 
task.  Nonetheless,  the  general  tramework  of  a  validation,  or  more 
precisely,  "simulation  confidence  building  procedures"  is  presented  in 
Section  ITT.  It  was  very  difficult  to  overcome  tb.e  temptation  to 
associate  this  approach  with  "documentation  alone."  While  the 
appendices  do  show  the  documentation  application,  the  concept  is  in  fact 
much  broader. 

Numerous  discussions  by  Working  Group  members  were  required  to 
develop  a  "unified"  understanding  of  the  process  itself  and  its  broader 
use.  Even  as  this  report  was  written,  some  members  felt  that  the 
process  may  be  too  complex  or  too  difficult  to  understand,  and 
therefore,  perhaps  will  never  be  used.  On  the  other  hand,  almost  all  of 
the  Working  Group  members  felt  that  a  prlmarv  activity  by  the  group  is 
to  "preach"  the  "process"  to  others  in  their  respective  countries. 
Publication  of  this  Advisory  Report  should  support  this  recommendation. 

Other  specific  aspects  of  missile  system  validation  were  covered 
fe.g.,  computer  languages,  verif Icatior./valldation/assessnent  methods) 
in  Sections  IV  and  V.  However,  these  treatments  were  of  necessity  quite 
limited.  Each  area  covered,  in  fact,  serves  as  a  topic  for  additional 
Working  Groups  or,  perhaps,  AGARDographs .  But  before  recommending  any 
specific  actions  of  this  type  to  the  Flight  Mechanics  Panel,  the  Working 
Group  felt  this  report  should  be  published  and  widely  circulated  within 
AGARD  for  comment  on  the  overall  topic.  Only  after  feedback  from 
experts  and  potential  users  should  additional  action  be  taken  by  the 
panel . 
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Fig  5.1  Interactions  Between  Verification,  Validation  and  Assessment 
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Fig  5.3  The  Software  Life  Cycle 
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Fig  5.6  Test  Systems 
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•  AUTOMATISATIQN  OF  TESTING  USING  TOOtS 

•  VALIDATION  OF  TEST  RESULTS  WITH  AUTOMATIC  COMPARISONS 

•  SECURITY  OF  THE  FUNCTION  USING  REDUNDANT  SOLUTIONS 


Fig  5.8  Dual  Program  Analysis  (NASA) 
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APPENDIX  A 


DEVELOPING  "CONFIDENCE  LEVELS  l^K  MODEL  BFPAVIOR" 
OR  THE  "CLIMB"  PROCESS 


CLIMB  LEVEL  I.  MODEL  SUMMARY,  PFSULTS  AND  CONCTSIONS 

Level  1  Includes  identification  of  tlie  mode!  d»'.e1oper,  summarv  of  the 
model,  and  general  description  of  the  model  wifi  -implilied  d’aprarrc. 

The  oblectlve  of  the  simulation,  don-.ain  of  interde'  app '  i  (  ;■  t  i  on  and 
criteria  for  model  validation  is  sta*^ed  Cite.  'l.e  nioue'’^  oritical 
variables  and  maior  assumptions  used  ir  im-dei  de. .  ’  i  pre-’t  are  identilied 
and  conclusions  on  overall  mode’  pertorm.nif  i  t<o. 

1.  MODEL  ORIGIN  AND  RFI.ATFP  I NKORMA  I  :  i '*■ 

1.1  Total  name  of  slmulatio.i  model 

1.2  Name  of  developing  <' rgan i 7a 1 1  or 

1.3  Address  of  organization 

1.4  Name  of  contact  for  addlticnial  inlormation  about  model 

1.5  Address  of  contact  for  model  Informat  ii'ii 

1.6  Telephone  number  of  contact 

1.7  Organization  for  which  model  was  developed 

1.8  Address  of  organization 

1.9  Contact  person 

1.10  Telephone  number  of  contact 

1.11  Keywords  for  data  base  processing 

?.  OBJECTIVES  IN  DEVELOPING  THE  SIiaiLATlON  MODEL 

2.1  Objectives  of  the  simulation 

2.2  Background  Information  leading  to  model  development 

3 .  MODEL  SUMMARY 

3.1  Definition  of  terms  (omitting  all  symbols) 

3.2  Conceptual  model  showing  major  Input/output  variables 

3.3  Summary  statement  and  descriptive  documentation  on  model 
application 

3.4  Nature  of  model  (Discrete,  Continuous,  Stochastic,  etc.) 

4.  FUNCTIONAL  MODEL 

4.1  Description  of  functional  model 

4.2  Simplified  functional  diagram  with  major  subsystem  and 
major  variables  identified 

4.3  Definitions  of  and  comments  on  major  variables  in 
functional  diagram 

4.4  Critical  variables  identified  for  model  validation 

5.  MODEL  APPLICATION 

5.1  Domain  of  Intended  application  of  simulation  model 

5.2  Major  assumptions  used  In  developing  the  model 


5.3  Major  knovm  limitation  in  domain  of  application 

5.4  Nonobvlous  exclusions  from  model 

5.5  Inputs  to  and  Outputs  from  Model 

6.  PHILOSOPHY 

6.1  Criteria  for  validation 

6.2  Methodology  for  validation 

7.  SUMMARY  COMMENTS  ON  SIMULATION  IMPLEMENTATION 

7.1  Type  computer  and  operating  system  for  simulation 

7.2  Computer  language  or  simulation  language  used 

8.  STUDIES  OK  AREAS  17HKRE  MODEL  HAS  BEEN  USED 

8.1  Specific  studies  where  model  was  used 

8.2  Related  model  background 

9.  COMMENTS  ON  MODEL  PERFORMANCE 

9.1  Summary  of  validation  results 

9.2  CLIMB  levels  achieved 

9.3  General  conclusions  on  model  performance 

10.  y^PPLICABLE  DOCUMENTS 


Cl  TMB  t,FV1:L  2.  SYSTEM  MODELS  AND  SUBMODELS  THEORETICAL  AND  INDIRECT 
DATA  BASES 


Simulation  model  and  submodel  performances  are  compared  with  theoretical 
models  and/or  existing  appropriate  validated  simulation  models.  Methods 
of  comparing  model  performances  are  identified  and  results  given  at  the 
level  of  visual  inspection,  expert  opinion  and  plot  overlays.  Analysis 
methodology  with  assumptions  and  deficiencies  are  identified. 

1  .  SY.'^TEM  MODEL  ELEMENTS 

1.1  General  description  of  system  model 

1.2  Block  diagram  of  system  model 

1.3  Identification  of  major  subsystems 

1.4  Assumptions  and  justifications  used  system  model 
development 

2.  IMPLEMENTATION  DESCRIPTION 

t 

2.1  List  of  computer  variables 

2.2  Processing  methods  used  and  relevant  parameter 
identification  (i.e.,  integration  methods,  initialization 
methods,  computer  word  length,  etc.) 

2.3  Required  program  library  elements 

2.4  Required  computer  resources 


3.  SYSTEM  MODEL  VERIFICATION 

3.1  Criteria  for  model  verification 

3.2  Identify  methods  used  for  model  verification 

3.3  Identify  data  base  used  for  model  verification 

3.4  Results  from  model  verification 

4.  VALIDATION  OF  SYSTEM  MODEL'S  STOCHASTIC  COMPONENTS 

1.1  Identification  of  stochastic  components 

4.2  Criteria  for  achieving  validation 

4.3  Validation  methods  and  techniques  used 

(Comparisons  of  means,  variances,  distributions,  etc.) 

4.4  Data  bases  used  for  validation 

4.5  Results  from  validation  effort 

5.  VALIDATION  AGAINST  OTHFR  EXISTING  MODELS 

5.1  Identification  of  existing  models  used 

5.2  Criteria  for  achieving  validation 

5.3  Validation  methods  and  techniques  used 

(Comparisons  of  means,  variances,  distributions) 

5.4  Data  bases  used  for  validation 

5.5  Results  from  validation  effort 

6.  SUBSYSTEM  CHARACTER! 7.ATI0N  AND  BRIEF  DESCRIPTION  OF  SUBSYSTEM  MODELS 

6.1  General  description  of  model 

6.2  Block  diagram  of  subsystem  model 

6.3  Criteria  for  validation 

6.4  Validation  methods  used 

6.5  Validation  results 

7.  BENCHMARK  TEST  CASE 

7.1  Description  of  benchmark  test  case 

7.2  Input  data  and  computer  configuration  for  test  run 

7.3  Output  data  or  sample  results  for  critical  variables 

7.4  Criteria  for  acceptabi 11  tv  of  benchmark  results 

8.  COMPUTER  PROGRAMS 

8.1  User  instructions 

8.2  Computer  listing 

9.  PROGRAM  VERIFICATION 

9.1  Criteria  for  verification 

9.2  Identify  methods  for  verification 

9.3  Identify  data  base  used  for  program  verification 

9.4  Results  from  program  verification 

10.  APPLICABLE  DOCUMENTS 
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CLIHB  LEVEL  3.  SUBSYSTEM  REAL  WORLD  DATA  BASF 

Real  world  data  Ip  usually  available  on  at  least  one  major 
subsystem  for  comparing  with  simulated  model  results.  Schematics  and 
technical  documentation  of  the  total  complex  model  is  included  only  if 
the  need  arises  resulting  from  the  validation  efforts  described  here. 
Statistical,  logical,  or  deterministic  methods  are  identified  for 
achieving  validation  of  the  subsystem  model.  Acceptability  of  the 
submodel  is  noted. 

1.  REAL  WORLD  SUBSYSTEM  DATA 

1.1  Identification  of  subsystem. 

1.2  List  of  variables  for  which  measured  data  exist.  (Real 
world  data  recorded  in  a  format  consistent  with  the 
format  of  the  simulated  generated  data.) 

1.3  Data  in  hard  copy,  i.e.  charts,  graphs,  plots,  etc. 
provided  to  correspond  to  the  critical  variables 
identified  in  CLIMB  LEVEL  1.  The  source  should  be 
identified  of  any  additional  data  available. 

2.  EXPERIMENTAL  TEST  ENVIRONMENT 

The  description  Includes  information  at  the  subsystem  level  not 
shown  in  CLIMB  LEVEL  2  diagrams,  i.e..  Inputs,  outputs,  test  points, 
scale  factors,  submodel  linkages,  etc. 

2.1  Scenario  used  to  excite  subsystem 

2.2  Description  of  test  experiment 

3.  METHODS  AND  TECHNIQUES  USED  TN  COLLECTING  REAL  WORLD  DATA 

3.1  Data  collecting  methods 

3.2  Error  sources  associated  with  input  and  output 
measurements 

3.3  Analysis  performed  on  input  and  output  measured  data 

3. A  Contact  for  further  information  on  measured  data 

-  Name 

-  Company  address 

-  Telephone  number 
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4.  THF  APPROACH  USED  FOR  VAI.TPATING  THE  SUBMODEL  USING  THE  REAL  WORLD 
DATA 

4.1  Criteria  for  validation 

4.2  Validation  methods  used 

4.3  Validation  results 

4.4  Model  changes  due  to  validation  effort 

5.  TECHNICAL  DOCUMENTATION  FOR  EXCITATION  MFTHOnS  fFXCITATlON  SOURCES 
MAY  BE  DIFFFPF.NT  FOR  SUBMODEL  AND  RFAT  WORLD  SUBSYSTEMS) 

5.1  Description  of  excitation  nethodfs) 

5.2  Documentation  of  excitation  data 

5.3  Documentation  of  real  world  subsystem  response 

5.4  Documentation  of  submodel  response 

5.5  Computer  program  listing  of  excitation  methods  (The 
computer  listing  of  the  submodel  simulation  will  be  shown 
if  different  than  the  listing  shown  in  CLIMB  LEVEL  2) 

5.6  Excitation  program  for  hardware  test  configuration 

6.  USER  INSTRUCTION  FOR  TEST  SET  UP 

7.  BENCHMARK  FOR  TEST  SET  UP 

7.1  Description  of  benchmark  test  ease 

7.2  Initial  conditions  for  test  set  up 

7.3  Output  data  or  sample  results  for  critical  variables 

7.4  Criteria  for  acceptable  benchmark  results 

8.  APPLICABLE  DOCUMENTS 


CLIMB  LEVEL  4.  HAPJIWARE-IN-THE-LOOP  OPERATION 

A  data  base  is  available  from  a  hardware-ln-the-loop  operation 
using  a  major  subsystem  hardware  component,  e.g.,  for  missile  systems, 
an  autopilot,  sensors/seekers,  embedded  computers,  actuators,  etc.  A 
typical  process  is  to  include  hardware  used  to  collect  data  from  CLIMB 
LEVEL  3.  Results  from  the  mode]  versus  hardware  performance  comparison 


M 

Is  reported  with  specifics  on  methods  used  for  data  comparisons  and 

performance  validation.  Included  are  specifics  on  any  additional  mode] 

development  environment  for  RF/EO/IR  seekers. 

1.  DESCRIPTION  OF  HARDWARE-IN-THE-LOOP  (IIV.’TL)  SYSTEM 

1.1  Description  of  hardware  to  be  used  for  HWIL  operation. 

1.2  Description  of  computer  system  (Analog,  Digital,  Hybrid) 
used  for  HWIL  operation.  Specifically,  was  the  all 
digital  simulation  program  partitioned  between  a  digital 
and  analog  computer? 

2.  PARTITIONED  MODEL  FOR  HWIL  OPERATION 

2.1  Diagram  of  the  partitioned  model  showing  elements  of  the 
model  to  be  replaced  with  hardware  (Hardware  Is  defined 
as  any  outside  element  connected  to  the  digital,  analog, 
or  hybrid  computer  relating  to  the  real  world  svstem.) 

2.2  Assumptions  and  criteria  for  selecting  the  particular 
partitioned  configuration  of  the  model. 

2.3  Model  variables  partitioned  between  the  digital  and 
connecting  systems.  Including  the  analog  computer. 

2.4  loentlfy  the  model  variables  showing  range  and  scale 
factors  for  the  connecting  systems. 

3.  RESULTS  OF  HARDWARE  IN  THE  LOOP  OPERATION 

3.1  Time  history  plots  of  critical  variables  showing  tlie 
all  digital  computer  results  and  HWIL  results  (Method  of 
showing  results  includes  means  and  variances  for  systems 
with  random  components.) 

3.2  Identify  any  data  analysis  performed  for  comparing  the 
all  digital  and  HWIL  results,  i.e.,  time  correlation 
analysis,  distribution  function  testing,  power  spectral 
density  testing,  etc, 

3.3  Identify  and  change  the  all  digital  simulation 
model  based  on  results  from  HWIL  operation. 

4 .  COMPUTER  PROGRAM 

4,1  Partitioned  digital  program 

4.1.1  Identify  changes  or  modlrication  to  the 

digital  computer  program  required  for  HWIL 
operation,  i.e.,  use  of  real  time  library 
subroutines,  integration  methods,  etc. 


4.1.2  Input  conditions  and  test  scenario  for 
executing  the  all  digital  simulation 
partitioned  for  HWIL  configuration. 

4.1.3  Expected  output  data  from  test  scenario 

4.1.4  Identify  special  program  development  for  HWTL 
operations,  l.e.,  real  time  data  recording, 
online  data  analysis  real  time  interrupt 
drivers 

4.1.5  List  special  programs  required  for 
real  time  or  HWIL  operations. 


4.2  Connecting  Systems 

4.2.1  Identify  the  critical  variables  between  the 
digital  program  and  connecting  systems. 

4.2.2  Identify  error  sources  associated  with 
connecting  system  variables. 


4.3  Model  Variables 

4.3.1  Show  verification  of  the  partitioned  model 
against  the  unpartitioned  model 

4.3.2  Show  verification  of  the  HWIL  system 
against  the  partitioned  model 


4.4  Model  Validation 

4.4.1  Validation  of  the  partitioned  model  against  the 
unpartitioned  model . 

4.4.2  Verification  of  HWIL  system  against  the 
partitioned  model. 

CLIMB  LEVEL  5.  TOTAL  REAL  WORLD  SYSTEMS  OPERATIONS 

A  data  base  Is  available  from  operating  the  total  real  world 
system.  As  a  minimum,  results  are  reported  on  the  validation  effort  for 
the  system's  critical  variables  operating  in  the  domain  of  intended 
application.  Specifics  on  validation  methodology  and  performance 
comparisons  are  reported.  Evaluation  and  conclusions  are  made  regarding 
the  system  model  performance  and  deficiencies. 


1. 


CONCLUSIONS  AND  COMMENTS  ON  MODEL  VALIDATION  EFFORT  USING  RFAI, 
WORLD  SYSTEMS  TEST  RESUITS 


?.  DESCRIPTIVE  SUMMARY  OF  REAL  WORLD  TEST  CONDITIONS  AMD  TES'I  EFSIILTS 

3.  SYSTEM  TFST  ENVIRONMENT 

3.1  Purpose  of  test 

3.2  Description  of  test  measurement  methodology 

3.3  Location  of  test  site  and  ambient  conditions  as  related  to 
system  tests;  i.e.,  temperature,  pressure,  wind  velocity, 
humidity,  etc. 

3.4  Error  tolerances  In  the  measurement  system 

4.  DESCRIPTION  OF  TEST  SCENARIO  USED  TO  STIMULATE  THE  REAL  WORLD 

SYSTEM 

4.1  Description  of  target  or  system  test  driver 

4.2  Target  or  test  driver  initial  conditions 

4.3  Method  used  for  reconstructing  system  test  driver.  (An 
example  for  a  missile  system,  reconstructing  the  target 
trajectory  would  be  required.) 

4.4  Reconstructed  system  test  driver  data 

5.  REAL  WORLD  SYSTEMS  PERFORMANCE  RECONSTRUCTION 

5.1  Initial  conditions  on  system  parameters  as  measured 

5.2  Method  of  system  performance  reconstruction 

5.3  Reconstructed  system  performance  data  (Example:  Missile 
position  and  velocity  history,  time  of  flight  to  closest 
approach,  position  of  closest  approach,  etc.) 

5.4  Measured  data  on  system's  critical  variables 

b.  STRUCTURING  OF  SIMULATION  MODFL  FOR  SYSTEM  TEST  CONDITIONS 

6.1  Identify  simulation  model  variables  Initialized  using  system 
test  data. 

6.2  Identify  assumptions  made  about  simulation  model  initial 
conditions  for  operating  with  system  test  conditions. 

6.3  Simulation  model  generated  data  using  real  world  system  test 
conditions. 


ANALYSIS  OF  SIMULATED  MODEL  GENERATED  DATA  AND  SYSTEM  TEST  RESULTS 


7.1  Identify  ciethodolrgy  of  data  comparison. 

7.2  Results  of  comparing  simulation  model  generated  data  and 
system  test  results. 


IDENTIFY  AND  EXPLAIN  DISCREPANCIES  BETWEEN  ADJUSTED  MODEL  GENERATED 
DATA  AND  REAL  WORLD  SYSTEM  TEST  RESULTS 


RECOMMENDATION  FOR  MODEL  IMPROVEMENT 


app?;kptx  b 


EXAMPI.FS  OF  THF  "Cl.IMB"  PROCESF. 

'Ilie  obiecFive  of  this  appendix  Is  to  Illustrate  the  use  of  the 
Confidence  Levels  in  Nodel  Behavior  fCLTMB)  process  using  as  examples  an 
actual  simulation  of  an  electrical  actuation  system  for  a  missile. 

These  examples  follow  the  outlities  of  CLIMB  Levels  1,  2  and  3  presented 

in  Appendix  A.  The  simulation  was  performed  at 

Messerschmi tt-Bolkow-Blohm  Cmbl!  under  the  code  named  FI.ACT  3. 


CLIMB  LEVEL  1  EXAMPLE 


1  .  MODFT.  ORIGIN  ANT)  RELAfFh  INFORMATION 

1 . 1  Total  Name  of  Simulation  Model 

ELACT3:  Electrical  Actuation  Svstem 


1 • 2  Name  of  Developing  Organiration 

Messerschmitt-Bol kow-BI ohm  CmbH 


1 . 3  Address  of  Developing  Organization 

Messer Schmitt -Bo Ikow-Bl ohm  CmbH 

Abteilung  AE13 

Postfach  801149 

D-8000  Munchen  FO 

W-Germany 

Telex:  5287-0  mbb  d 


1 .4  Address  of  Contact  for  Model  Information 

Person:  Werner  Bub 
Address;  same  as  Paragraph  1.3 
-  Phone  No:  089-60004125 


1.5  Address  of  Contact  for  Additional  Information  About  Model 


Persons: 

Frldbert  Kilger,  Phone  No  089-60002302 
Herman  Neubauer,  Phone  No  089-60006364 


Address : 


Same  as  para  1.3 

1 .  t>  Organization  for  Which  Model  was  Developed 

Same  as  para  1.3 

1  . 7  Contact  Person 

Person: 

Alfred  Huber,  Phone  No  089-600C0815 
-  Address: 

Same  as  para  1.3 

1 . 8  Keywords  for  Pata  Base  Processing 

Electrical  Actuator,  Missile  Simulation 

2.  OBJECTIVES  IN  DCVELOPTMC  THE  SIMULATION  MODEL 

2. 1  Objectives  of  the  Simulation 


In  tactical  missile  systems,  a  set  of  four  fins  moved  by  actuators 
usually  constitute  the  control  surfaces  of  the  missile  airframe.  These 
are  contained  in  the  autopilot  loop  to  produce  the  three  rotational 
degrees  of  freedom  of  the  missile. 

Objective  of  the  development  of  the  present  model  was  to  provide  a 
subsystem  model  of  an  electrical  actuator  system,  taking  hinge  moments 
into  account,  that  could  be  included  in  an  overall  missile  system 
simulation  model. 

2 • 2  Background  Information  Leading  to  Model  Development 

The  Model  was  developed  in  1978  to  be  used  for  simulation  of  the 
EMS  Experimental  Missile  System. 


3.  MODEL  SUMMARY 

3.1  Definition  of  Terms 


Commanded  fin  Input  variable,  fin  deflection  demanded  from  the 

deflection  autopilot  system 


External  hinge  Input  variable,  hinge  moment  generated  by  the 

moment  aerodynamic  fin  forces  and  moments 


L 


Device  locked 

Actual  fin  de¬ 
flection 

Motor  speed 

Commanded  motor 
current 

Actual  motor 
current 


Motor  feeding 
voltage 


A 
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Logical  Input  variable,  if  true,  the  actuator 
Is  locked  in  its  initial  position 

Output  and  state  variable,  actual  angular  posi¬ 
tion  of  the  shaft  on  which  the  fin  is  mounted 

Output  and  state  variable,  angular  speed  of  the 
shaft  of  the  electrical  motor 

State  variable,  output  variable  of  the  actuator 
controller 

Output  and  state  variable,  actual  motor  current 
generated  in  response  to  demand  from  actuator 
controller 

Output  variable,  voltage  across  the  terminals  of 
the  electrical  actuator  motor 


Conceptual  Model  Showing  Major  Input /Output  V'ariables 


3.2 


Actual  fin  deflections,  motor  speed,  motor  voltage  and  motor  current  are 
computed  as  a  function  of  commanded  values  for  desired  fin  deflections, 
the  moments  actually  acting  on  the  hinges,  and  a  logical  variable  which 
determines  whether  the  device  is  locked  or  unlocked.  I'ig  I  shows  the 
basic  functions  which  model  ELACT3  performs. 


commanded  fin  deflectloii 

actual  fin  deflection 

1 

1 

hinge  moment 

FI.ACT3 

motor  speed 

locked /unlocked 

1 

1 

! 

motor  current 

Fig  B-1  Basic  Model  Functions 


3. 3  Summary  Description  of  Model  Application 

Model  ELACT3  car  be  used  in  the  scope  of  missile  models  if  a  mode] 
of  the  aerodynamic  hinge  moments  acting  on  the  fins  is  available. 
Intended  applications  are; 

-  Autopilot  studies 


-  System  simulation  studies  -  if  hinge  moments  have  a  sensible 
effect  on  system  performance  or  If  an  estimate  of  overall  power 
consumption  during  a  mission  has  to  be  obtained 

-  Actuator  design  studies  for  verification  of  basic  design 
parameters 

-  Usage  as  a  model  of  a  typical  electrical  actuator  svstem  for 
other  applications  where  load  moments  are  important 


3.4  Nature  of  Model 


The  model  is  continuous,  l.e.,  it  is  described 
differential  equations.  The  model  is  basically  of 
in  the  sense  that  it  does  not  contain  any  internal 


hy  three  ordinary 
deterministic  nature 
sources  of  noise. 


4. 


FUNCTTONAI,  MODEL 


A .  1  Description  of  Ftircflonal  Model 

Since  the  Influence  of  external  hinge  iroinents  is  taken  into 
account,  a  physically  functional  model  Is  necessary.  Therefore,  the 
model  Is  composed  of  the  actuator  controller,  the  electrical  power 
amplifier,  the  electrical  shunt  motor  with  gear  drive,  and  pickups  for 
motor  speed  and  fin  position.  The  dynamics  of  the  power  amplifier  as 
well  as  the  dynamics  and  higher  order  effects  of  the  rotor  circuit  are 
neglected.  The  overall  dynamics  for  small  signals  corresponds  to  a 
third  order  transfer  function.  Nonlinear  behaviour  is  the  result  of 
limits  for  motor  current,  motor  voltage,  and  motor  speed,  which  are 
represented  in  the  model. 

4 . 2  Functional  Block  Diagram  and  Major  Variables 

The  model  is  composed  of  the  iol lowing  functional  blocks: 

-  Actuator  controller 

-  Electrical  power  amplifier 

-  Flectrlcal  motor  with  associated  gear  drive 

-  Sensors  for  fin  position  and  motor  speed 

The  relationship  between  these  function  of  blocks  are  depicted  in 
rbe  following  functional  block  diagram: 


voltage  Current  speed 


Fig  B-2  Functional  Block  Diagram 


4.3  Definition  and  Comments  on  Major  Variables  in  Functional 

Block  Diagram 


Same  as  in  Paragraph  3.1 


4.4 


Critical  Variables  for  Model  Validation 


-  Actual  fin  deflection 

-  Motor  speed 

-  Actual  motor  current 


5.  MODEL  APPLICATION 

5 . 1  Domain  of  Intended  Application  of  Simulation  Model 

The  model  can  be  used  without  special  precautions  within  the  domain 
defined  by  its  basic  design  parameters  fmax  fin  deflection,  max 
deflation  rate,  max  binge  moments,  bandwidth,  etc.). 


Tr  view  of  the  real  actuator  system  and  of  the  intended 
applications,  the  model  represents  the  following  features: 

-  Third  order  dynamics 

-  Motor  current 

-  Friction  and  hinge  moments 

-  Limitations  in  actual  fin  deflection,  motor  speed  and  motor 
current 


-  Uigld  body  dynamics 


If  the  design  of  the  real  actuator  system  is  sound,  the  neglected 
effects  such  as  backlash,  gear  efficiency,  elasticity  of  mechanical 
parts,  motor  commutation  and  cogging  effects,  deterioration  of  magnetic 
flux  and  the  dynamics  of  the  power  amplifier  should  not  have  a  sensible 
effect  on  the  static  and  cyuamlc  behavior  of  the  device  and  therefore 
also  on  the  model . 

5.4  Non-obvlous  Exclusions  from  Model 


An  autopilot  model  which  calculates  desired  fin 
deflections 
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A  mode]  of  hinge  moments  which  computes  the  aerodynamics 
load  moments  acting  on  the  actuator  hinge  as  a  function 
of  fin  Incidence. 

5.5.2  Models  Using  Outputs 


The  principal  output  of  model  ELACT3  Is  actual  fin  deflection. 
This  output  provides  data  to  compute  aerodynamic  forces  and  moments 
acting  on  the  missile  body  and  on  the  actuator  hinges.  Additional 
outputs  can  be  used  to  monitor  actuator  performance.  The  output  motor 
current  in  conjunction  with  a  power  supply  model  can  be  used  to 
determine  power  consumption  over  missile  flight  time. 

6.  MODEL  VALIDATION  PKII.OSOPHY 

6.1  Criterion  for  Validation 


Criterion  for  validation  requires  that  the  model  response  and  the 
response  of  the  real  actuator  system  be  matching  reasonably  well  from  an 
engineering  point  of  view  using  the  same  kind  of  system  excitation  and 
observing  the  variables  Identified  in  Paragraph  h.U. 

6. 2  Methodology  for  Validation 

Validation  was  performed  against  data  obtained  from  bench  test  with 
the  real  actuation  system.  A  step  function  for  "commanded  fin 
deflection"  was  applied  as  an  Input  test  function.  The  system  response 
with  respect  to  the  critical  variables  Identified  In  Paragraph  A. A  were 
recorded  on  a  multi-channel  recorder.  The  corresponding  test  was 
performed  with  the  model  and  the  critical  variables  v/ere  recorded  on 
plots  using  the  same  format  and  scale  factors  as  on  the  multi-channel 
recorder.  Comparison  was  performed  by  visual  overlay  of  the  two  system 
responses.  Quality  of  coincidence  was  judged  by  engineers  experienced 
in  actuator  design  and  In  missile  modelling.  Mo  formal  measures  for 
goodness  of  fit  have  been  used. 


7.  SUMMARY  COMMENTS  ON  SIMULATION  IMPI  FKENTATION 


7.1 

NDS 

7.2 


Type  Computer  and  Operating  System 

The  Model  is  Implemented  digitally  on  an  CDC  6600  Computer  under 
l.A,  level  552. 

Language 

Standard  ANSI-FORTRAN  IV. 


8.  STUDIES  OP.  AREAS  WHERE  MODEL  HAS  BEEN  USED 

8. 1  Specific  Studies  where  Model  was  used 

The  model  was  used  for  the  purpose  mentioned  in  Paragraph  2.2. 


1 


F.elated  Model  Background 
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8.2.1  Similar  Models 


Model  EI.ACT.I  is  a  member  of  a  family  of  several  actuator  models: 

-  ACTl;  first  order  general  actuator  model  with  position 
and  speed  limits. 

ACT?:  Second  order  general  actuator  model  with  limits 

for  acceleration,  speed  and  position. 

EI.ACTj:  Third  order  electromechanical  actuator  model, 
taking  Utr.lts  and  hinge  moments  into  account. 

-  EhACT4:  Detailed  model  to  be  used  in  electrical 

actuator  design  studies. 

8.2.2  Mode]  Structure 

Model  ELACT3  is  a  stripped  version  of  model  ELACT4  that  used  for 
design  of  the  actuator.  The  newly  developed  actuator  system  was 
acceptance  tested  against  results  obtained  with  ELACT4.  In  ELACT3,  only 
those  features  are  represented  that  are  necessary  to  meet  the  objectives 
mentioned  In  Paragraph  2.1.  The  choice  has  been  made  by  engineers 
experienced  in  actuator  design  and  in  missile  system  modelling. 

8.2.3  Model  Data 


The  parameters  and  constants  for  the  model  have  been  taken  from 
model  FLACT4  and  have  been  validated  bv  measurements  on  the  actual 
system  during  its  development. 

9.  COMl'ENTS  ON  MODEL  PEKFOKMANCE 


9.1  Summarv  of  Validation  Results 


Since  no  device  was  available  which  would  be  capable  to  applv  a 
defined  moment  on  the  hinge  of  the  real  system  under  dynamic  conditions, 
validation  was  possible  only  without  external  load. 

For  the  tests  performed,  coincidence  of  the  variables  "fin 
deflection"  and  "motor  speed"  was  very  good  whereas  the  motor  current  of 
the  model  matched  the  current  of  the  real  system  reasonably  well  onlv 
during  acceleration  and  deceleration  phases.  A  large  ripple,  which  is 
induced  in  the  real  system  by  motor  effects  such  as  cogging, 
commutation,  etc.,  does  not  exist  in  the  case  of  the  model  since  motor 
effects  are  not  Included. 

The  way  mechanical  friction  was  Tepresented  in  the  model  v;as  not 
reasonable.  VTien  the  model  approached  a  steady  state,  a  limit  cycle 
was  generated;  the  characteristics  of  which  are  very  sensitive  to  the 
Implementation  parameters  (e.g.,  integration  step  size). 


9 . 2  CLIMB  Level  Achieved 

CLIMB  Level  3  has  been  achieved. 

9. 3  General  Conclusions  on  Model  Performance 

ELACT3  Is  a  reasonable  model  of  a  third  order  actuation  system. 

The  static  and  dynamic  performances  are  well  represented.  The  represen¬ 
tation  of  the  motor  current  In  the  model  allows  the  correct  represen¬ 
tation  of  degradation  In  dynamic  performance  when  the  current  reaches 
its  limitation  bounds  as  well  as  to  obtain  an  estimate  of  electrical 
power  consumption,  whereas,  the  representation  of  motor  current  with 
respect  to  time  Is  poor  because  of  the  neglected  high  rrcer  effects. 
Caution  has  to  be  observed  when  using  the  mechanical  friction  feature  of 
the  model,  as  explained  In  Paragraph  9.1. 

When  the  model  Is  used  within  an  autopilot  loop,  steady  state 
conditions  will  practically  never  he  reached  and  the  limit  cycle  will 
probably  never  be  excited.  Therefore,  If  one  wishes  to  derive  an 
estimate  of  power  consumption  of  the  actuation  system,  the  model  could 
be  used  taking  friction  Into  account  If  the  necessary  caution  Is 
observed . 


10.  APPLICABLF  DOCUMENTS 

ELACT4,  Iiocumentation  of  the  Design  Model  of  an 
Electrical  Actuation  System. 


CLIMB  LEVEL  2  EXAMPLE 
1.  SYSTEM  MODEL  ELEMENTS 

1 . 1  General  Description  of  System  Model 

The  model  is  composed  of  the  following  functional  blocks: 

-  Actuator  controller 

-  Electrical  power  amplifier 

-  Electrical  motor  with  associated  gear  drive 

-  Sensors  for  fin  position  and  motor  speed 
These  can  readily  be  Identified  In  Fig  B-3. 

1 . 2  Block  Diagram  of  System  Model 

Represented  by  Fig  B-3 

1 . 3  Major  Subsystems 


Oetniled  Block 
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1.3.1  Actuator  Controller 

The  actuator  controller  is  a  PID-controller ,  the  three  coefficients 
of  which  are  calculated  from  the  denominator  polynomial  of  the  desired 
third  order  overall  transfer  function. 

1.3.2  Power  Amplifier 

The  power  amplifier  provides  a  current  to  the  motor  that  Is 

commanded  by  the  actuator  controller.  Neglecting  dynamic  effects,  it  is 
represented  by  Its  steady  state  behavior  taking  into  account  limits  for 
motor  current  and  voltage. 

1.3.3  Flectrlcal  Hotor/Gear/l.oad 


The  model  of  this  block  uses  the  basic  laws  of  a  dc  shunt  motor, 
neglecting  the  dynamics  of  the  rotor  circuit.  The  gear  is  represented 
by  its  ratio  and  Its  coefficient  for  friction. 

1.3.4  Sensors 


The  pickups  for  motor  speed  and  fin  deflection  are  modelled  by 
error  terms  for  set-off  and  scaling  errors. 


1.4 

Model 

Interface 

1.4.1 

Model 

Inputs 

Mnemonic 

Name 

Type 

Symbol 

Dimension 

Meaning 

TX 

LOGICAL 

T 

“T.. 

- 

Device  locked 

If  true 

DT 

REAL 

- 

s 

Comm.unlcation 

interval 

STC 

REAL 

0 

c 

rad 

Commanded  fin 

deflection 

MH 

REAL 

"'h 

Nm 

Hinge  moment 

Tab  1  •  Inputs  to  ELACT3 
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Model  Outputs 


1.4.? 


Mnemonic 

Name 

Type 

Symbol 

Dimension 

Meaning 

SI 

REAL 

0 

rad 

Fin  position 

DSI 

REAL 

rad  /  s 

Motor  speed 

IC 

REAL 

1 

c 

A 

Commanded  motor  current 

TM 

REAL 

A 

Actual  motor  current 

UC 

REAL 

u 

c 

V 

Motor  feeding  voltage 

Tab  2  Outputs 

from  ELACT3 

1.5 

Assumptions  and  Justifications 

1'sed  for  System  Model 

See 

Paragraph 

10,  B-1. 

1.6 

Mathematical  Model 

In  the  following  paragraphs,  the  ma thenar  leal  model  of  the 
electromecharlcal  actuation  system  will  be  described.  The  model 
variables  are  listed  in  Tab  3,  the  model  constants  and  parameters  are 
listed  In  Tab  4,  and  the  detailed  Mock  diagram  of  the  system  Is  shown 
In  rig  B-3. 

The  following  conventions  have  been  used  for  notation: 

Constants  and  parameters;  Capital  letters 
Variables:  Small  letters 

Subscript  "M"  stands  for  "Kotor" 

Subscript  "m"  stands  for  "Measured  value" 

-  Subscript  "c"  stands  for  "Commanded  value” 

"V"  Indicates  "value  of  limitation  for  variable  v" 


Mnemonic 

Name 

Type 

Symbol 

Dimen¬ 

sion 

Meaning 

Remarks 

IC 

REAL 

1 

c 

A 

Cofomanded  motor 

current 

State  variable 

SIC 

REAI. 

a 

c 

rad 

Commanded  fin 
deflection 

Input  variable 

STM 

REAL 

0 

m 

rad 

Measured  fin 
deflection 

DSI 

REAL 

°  M 

rad/s 

Motor  speed 

Srate/output  var. 

DSIM 

REA!, 

m 

rad/s 

Measured  motor 
speed 

TM 

REAI. 

A 

Actual  motor 

Sr.ate/output  var. 

UC 

REAL 

“c 

V 

Motor  feeding 
vol tage 

Output  variable 

UE 

REAL 

“e 

V 

Motor  EhfF 

ST 

REAL 

0 

rad 

Actual  fin  de¬ 
flection 

State/output  var. 

LL 

LOGICAL 

logical 

"Device  locked" 

Input  variable 

MM 

REAL 

Nm 

Motor  moment 

MH 

REAL 

Nm 

External  hinge 
moment 

Input  variable 

Tab  3.  Model  Variables 
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Mnemonic 

Type 

Symbol 

Dlmen- 

Value 

Toler- 

Def Inl- 

Meaning 

Name 

slon 

ance 

tion  by 

OM 

REAL 

u 

1/s 

185 

- 

DATA 

System  design  parameter 

B 

REAL 

b 

- 

3.1 

- 

DATA 

System  design  parameter 

C 

REAL 

c 

- 

4.0 

- 

DATA 

System  design  parameter 

KG 

REAL 

- 

86 

- 

DATA 

Gear  ratio 

K1 

REAL 

As 

b  J  .'/C„  ±  67 

M 

DATA 

Controller  parameter 

K2 

REAL 

K, 

s 

c//b  (o) 

±  67 

DATA 

Controller  parameter 

K3 

REAL 

I/s 

6i/b 

t  67 

DATA 

Controller  parameter 

K4 

REAL 

- 

1 

* 

DATA 

6-sensor,  scaling  error 

K5 

REAL 

S 

- 

1 

t  0.5% 

DATA 

o-sensor,  scaling  error 

DELSI 

REAL 

Ac 

rad 

0 

i  0.006 

DATA 

o-sensor,  set  off  error 

SDMAK 

REAL 

rad/s 

288 

i  57. 

DATA 

Controller,  limit 

SIMAX 

REAL 

'o 

rad/s 

0.349 

t  57 

DATA 

Limit  of  commanded  fin 
deflection 

RM 

REAL 

0 

5.4 

5.4. .6.8 

DATA 

Motor  resistance 

CE 

REAL 

Vs/rad 

0.0707 

±  10% 

DATA 

Coefficient  of  EMF 

CM 

REAL 

Nm/A 

0.0707 

t  107 

DATA 

Motor  constant 

IMG 

REAL 

.T 

Nms^ 

10-' 

t  107 

DATA 

Moment  of  inertia 

ICMAX 

REAL 

A 

6.76 

i  5% 

DATA 

Limit  commanded  motor 

current 

t'CMAX 

REAL 

IT 

V 

56 

DATA 

Power  amplifier  max. 

output  voltage 

CR 

REAL 

s 

- 

0.2 

t  507 

DATA 

Coefficient  of  friction 

MR 

PEAL 

”r 

Nm 

0.P2 

t  507 

DATA 

Friction  moment 

MHMAX 

REAL 

Nm 

"h  ■'c 

DATA 

Max.  hinge  moment 

Tab  4.  Model  Parameters  and  Constants 


I 
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1.6.! 


Actuator  Controller 


6M 


The  differential  equation  for  the  PTH-cortroller ,  taking  the  gear 

us: 

Ko  (o. 


ratio  into  account.  Is  as  follows: 


with  the  limits 


-  "i  •  "3 


«.>  -  1 


Mm 


K 


'  Mm 


If  h,  =  true  (device  locked):  1  =0. 

L  c 

In  the  real  system  g„  is  generated  by  a  differentiating  network 
(Ref  Fig  B-3).  For  the  model,  Is  generated  from  the  equilibrium  of 

moments.  Ref  Paragraph  1.6.3.  Oj^,  is  formed  by  multiplication  with  K,, 
Ref  Paragraph  1.6.4.  ™ 


1.6. 1.1  Computation  of  Controller  Parameters 

The  overall  transfer  function  of  the  actuator  system,  neglecting 
the  dynamics  of  the  motor  current  circuit.  Is: 


1 

o(s)  =  "■  •  o  (s) 

.1  3  K,  2  ^ 

■  ■  -- .  #  s  +  ~  0  s  *  — —  +  1 

KiK2C^.  K3  K3 

Given  the  transfer  function  of  the  desired  behavior  of  the  actuator 
system 

o  (s) 
c 


u  m  (ij 


we  get  by 

comparison  of 

coefficients: 

’'l 

.  2 

=  b  •  (i)  « 

J/C^ 

"2 

=  c/(b  m) 

*0 

3 

It 

o(s)  = 


(^)^  +  c 


(£)? 


+  b 


(-) 


+  1 


J  Is  the  total  moment  of  Inertia  of  the  motor /gear/fin  assembly, 
defined  at  the  motor  side  of  the  assembly.  This  way  It  Is  possible  to 
calculate  the  coefficients  of  the  actuator  controller,  given  the  dynamic 
design  parameters  u,  b,  c. 


7(1 


1.6.?  Power  Ampliflpr 

The  task  of  tlie  power  amplifier  is  the  t  ransf orma t i <'n  of  the 
commanded  motor  current,  as  computed  hy  the  actuator  controller,  irtt' 
real  motor  current.  The  bandwidth  of  the  real  device  is  around  600  11  . 
This  is  large  compared  with  the  bandwidth  of  the  overall  actuator  svsiero 
and  can  therefore  certainly  be  neglected  for  the  intended  purpose  of 
this  model. 

The  steady  state  equations  of  the  power  amplifier  arc. 

"  ^E  *  ”11 

’V  =  ‘c  • 

If  |u  1>U  :  u  =1’  *  sign  (i  ) 

I  c  I  c  c  c  c 

1.6.3  Mo tor /Pear /Load 

Homent,  generated  by  the  motor: 

“m  =  S*  -M 

Equilibrium  of  moments: 

'^''m  '  *  "’l  ■  S*  {"'Ll* 

"’r,  =  "’h^^g 


=  /  K^)dt  +  0^ 


I f  L,  =  true  (device  locked):  o..  =  c„  =  0 
I  M  Mo 

0  =  0  =  Ao 

o 


Sensors 


The  feedback  values  for  the  controller,  motor  speed  and  actual  fin 
deflection,  are  measured  by  a  taebo  generator  and  a  potentiometer.  The 
tacho  generator  is  represented  by  a  scaling  error: 

c„  =  K,  •  o,, 

Mm  _  4  M 

oMro  "  ^4  *  '"m 

The  position  pickup  is  represented  by  set-off  and  scaling  error: 


om  =  •  o  -  Ao 


2.  IMPLEMENTATION  DESCRIPTION 

The  model  Is  Implemented  digitally  as  a  single  subroutine  named 
FLACT3,  written  In  Standard  ANSI  FORTRAN  IV. 

2. 1  List  of  Computer  Variables 

The  computer  variables,  apart  from  temporary  variables,  are  listed 
in  the  following  tables: 


Input  Variables  Tab  1 
Output  Variables  Tab  2 
Model  Variables  Tab  3 
Model  Parameters  and  Constants  Tab  4 
Implementation  Variables  Tab  5 
Implementation  Parameters  Tab  6 


Mn emonlc 

Type 

Pef inltlon 

by 

Meaning 

LL 

LOGICAL 

Input 

true  =  initialization 
false  =  Integration 
identical  to  "device 
locked" 

TERR 

INTEGER 

condition  within 
ELACT  3 

Error  Indicator 

TP, 

INTEGER 

ELACT  3 

Counter  for  integration 
control 

Tab  5. Implementation  Variables 

Mnemonic 

Type 

Definition 

by 

Meaning 

DT 

REAL 

input 

Communication  Interval 

DTRM 

REAI, 

ELACT 3 

Internal  Integration  step 

DTRMO 

REAL 

DATA 

Upper  bound  for  Integration 
step 

IRM 

INTEGER 

ELACT 3 

Number  of  Integration  steps 
per  communication  Interval 

TAB  6.  Implementation  Parameters 


X 


72 


i 


2 . 2  Processing  Methods 

2.2.1  Function  Allocation 

The  functions  are  performed  as  described  In  Paragraph  1.6.  There 
Is  a  clear  correspondence  between  the  Implemented  code  and  those 
funct 1 ons . 

2.2.2  Integration  Method 

For  Integration  of  the  differential  equations,  the  Euler  method 
Is  used.  Because  of  the  dynamics  of  the  modeled  device,  usually  a 
smaller  integration  step,  DTRM,  than  the  communication  interval  DT  has 
to  be  used.  During  Initillzatlon,  RLACT3  computes  a  suitable 
integration  step  size,  assuring  that  It  is  an  integral  fraction  of  the 
communication  Interval  not  greater  than  DTRMO: 

DTRM  =  DT/n  <  DTRMO 

with  n  =  suitable  Integer: 

n  =  Int  [DT/DTRMO  +  0.5] 

At  every  call  to  EhACT3,  n  Integration  steps  are  performed.  This 
Is  controlled  through  the  variable  IR  and  parameter  IRM  (=  n) . 

2.2.3  Model  Parameters  and  Constants 

Model  parameters  and  constants  are  defined  by  DATA-statements 
within  EhACT3.  They  cannot  be  altered  by  calling  the  subroutine.  There 
Is  no  stochastic  variation  of  model  parameters  Implemented. 

2.2.4  Initialization 

The  first  call  to  ELACT3  must  be  an  Initialization  call.  This  Is 
performed  by  calling  it  with  the  input  variable  LI.  -  TRUE  .  State 
variables  are  set  to  Initial  conditions,  the  controller  coefficients  are 
computed  as  well  as  the  Internal  Integration  step  size,  and  the  error 
Indicator  lERR  is  reset.  Initialization  calls  can  be  repeated. 

2.2.5  Error  Detection 

The  error  Indication  Is  set  to  1  if  the  external  hinge  moment 
exceeds  807  of  the  maximum  hinge  moment  defined  as 

2 . 3  Required  Program  Library  Elements 

Apart  from  standard  run  time,  library  routine  ELACT3  does  not  call 
any  subroutine. 

2.4  Required  Computer  Resources 


L 


The  program  does  not  need  any  peripheral  equipment  or  data  files. 


2.4.2  Memory  Requirements 

Subroutine  EI-ACT3  occupies  152  60-blt-words  of  main  memory  on  CDC 

6600. 

2.4.3  Running  Time 

The  execution  time  per  call  to  ELACT3  with  5  Internal  Integration 
steps  on  a  CDC  66000  Computer  Is  480  sec. 

3.  SYSTEM  MODEL  VERIFICATION 

3. 1  Criteria  for  Model  Verification 

The  following  criteria  have  been  used  for  model  verification: 

(a)  Model  responses  to  step  Input  functions  should  be  as  predicted 
by  theory  and  as  expected  due  to  an  expert's  understanding  of 
the  system  (plausibility). 

(b)  Insensitivity  of  model  behavior  with  respect  to  digital 
Integration  parameters. 

3 . 2  Methods  Used  for  Model  Verification 

The  methodology  used  for  model  verification  Is  summarized  in  Tab  7. 


3.2.1  Dynamic  Model  Behavior  Test 

Objective  was  to  assure  that  the  model  behaves  dynamically  as 
expected  from  theory  and  from  an  experts'  experience. 


3.2.2 


Sensitivity  with  Respect  to  Implementation  Parameters 


Objective  was  to  assure  that  model  behavior  does  not  depend  on 
Implementation  particularities  and  parameters. 

3. 3  Data  Bases  used  for  Verification 

3.3.1  Dynamics  Model  Behavior  Tests 

3. 3.1.1  Reference  Data  Base 

3. 3. 1.1.1  Model  Behavior  without  Mechanical  Friction 

Neglecting  mechanical  friction  (M^^  =  Cj^  =  0),  the  following  tests 
have  been  performed. 


Test  A: 


Test  B; 


Test  C: 


Operation  in  nonlinear  domain. 

Rectangular  input,  c  =  ±  0.26  rad,  no  external  load, 
change  in  a  every  aBo  ms. 

The  transitfon  slope  of  o  is  required  to  correspond  to 
maximum  motor  speed  divided  by  gear  ratio: 

r.  =  2P8/96  =  3  rad /sec 
Mmax 

Therefore,  neglecting  the  dynamics  of  the  system,  the 
transition  time  of  o  between  steady  state  values  should 
be  approximately 

t  =  ta/6^  =  0.52/3  =  0.173  sec 

Max 

The  motor  current  i^^^  is  required  to  overcome  mechanical 
inertia  during  the  acceleration/deceleration  phases.  As 
soon  as  m.otor  speed  n  reaches  its  saturation  value  of 
288  rad/sec  or  the  value  zero,  motor  current  Ij^  has  to  go 
to  zero. 


Operation  at  the  limits  of  linearity. 

Rectangular  input,  a  =  i  0.07  rad,  no  external  load, 
change  in  every  400  ms. 

Operation  in  the  linear  domain. 

Rectangular  input,  a  =  ±  0.017  rad,  no  external  load, 

change  in  a  every  400  ms. 
c 

The  required  model  response  can  be  calculated 
analytically,  using  the  desired  transfer  function  (see 
Paragraph  1.4. 1.1): 


'  OJ  'oi' 


- - - c  (s) 

+  c.-^  +  1 


with  parameters  u,  b,  c,  according  to  Tab  4. 


For  this  small  step  input,  no  one  of  the  limitation  values  is 
reached  and  the  implemented  model  should  reproduce  the  analytical 
solution  with  high  accuracy. 

Test  D;  Nonlinear  operation  with  external  hinge  moment.  Rectangular 
input,  a  =  +  0.26  rad,  m„  =  -15  Nm,  change  in  o  every 

400  ms. 

As  long  as  neither  currents  nor  voltages  reach  their 
saturation  values,  the  time  histories  of  o  and  should  be 
the  same  as  with  Test  A.  Also  1^^  should  show  the  nehavior 
as  in  Test  A,  but  with  a  constant  offset  value  which  is 
necessary  to  compensate  for  the  external  load.  This  offset 
value  should  be 


3. 3. 1.1. 2  Model  Behavior  Including  Mechanical  Function 

Above  Tests  A  through  D  have  to  be  repeated  with  mechanical 
friction,  l.e.,  =  0.02  Nm  and  =  0.2.  In  test  cases  without 

external  load,  the  behavior  of  o  and  should  be  Identical  to  the 
results  without  friction.  During  non-steady  states,  motor  current 
should  be  Increased  by 

M 

R  0.02 

A1  =  —  •  sign  =  - •  sign  (o  )  =0.28  •  sign  (c  ) 

C„  0.0707 

r 

Tn  test  case  D  (with  external  load)  the  motor  current  Is  increased  by 

'“r  ,  "b  •  c,/K^  ,  i>.o;.i5.o.2W 

Al  - - sign  In  )  - - ^Ipn  (c  )=0.725  .  sign  (o 

C,  ”  0.0707  '  '' 

M 

3.3. 1.2  Model  Generated  Data  Rase 

The  Integration  step  size  used  was  1  ms.  The  data  Is  recorded  In 
form  of  plots  in  a  format  similar  to  the  one  obtained  from  a  multi¬ 
channel  recorder.  The  first  line  shows  the  input  variable  r  ,  the 
following  one.s  the  critical  variables  defined  In  Section  10,  B-1. 

3.3. 1.2.1  Model  Behavior  without  Mechanical  Friction 

Test  A:  Operation  In  nonlinear  domain.  Fig  B-4  (At  end  of  CLIMB 

Level  2) . 

Test  B:  Operation  at  the  limits  of  linearity.  Fig  B-5. 

Test  C:  Operation  in  the  linear  domain.  Fig  P-6. 

Test  D:  Nonlinear  Operation  with  external  hinge  moments.  Fig  B-7. 

3.3. 1.2.2  Model  Beha\’lor  Including  Mechanical  Friction 

Test  A:  Operation  In  the  nonlinear  domain.  Including  mechanical 

frlctloti,  l.e.,  M  =  0.0?  and  C  =  0.02,  Fig  B-P. 

K  1  • 

3.3.2  gensltlvity  v;l tb  Respect  to  Implementation  Parameters 

Using  Test  C  as  test  case,  the  sensitivity  of  model  response 
with  respect  to  Integration  step  size  DTRM  has  been  investigated. 


3.3.2.  1 


Reference  Data  Base 


Test  C:  Operation  In  the  linear  domain,  without  mechanical 

friction,  analytical  solution.  Fig  B-9. 


3. 3. 2. 2 


Model  Generated  Data  Base 


Test  C; 


Operation  In  the  linear  domain,  without  mechanical 
friction,  results  of  runs  with  DTRM  =  0.0005  s,  0.001  s 
and  0.002  s.  Fig  B-10. 


Test  C: 


As  above,  but  with  DTRM  =  0.002  s,  0.003  s  and  0.004  s. 
Fig  B-11. 


Using  the  same  test  case,  the  model.  Including  mechanical  friction 
was  exercised.  Fig  B-12  shows  the  results  in  the  case  of  DTRM  =  0.001 
s.  Fig  B-13  with  DTRM  =  0.002  s. 

3 . 4  Results  from  Model  Verification  Efforts 

3.4.1  Dynamic  Model  Behavior  Tests 

Criterion  was  reasonably  coincidence  between  the  two  data  bases 
from  an  engineering  expert's  viewpoint.  Plot  overlays  have  been  used, 
but  no  quantitative  measures  for  goodness  of  fit  were  used.  Tn 
addition,  the  model  generated  data  did  not  expose  any  anomalies  or 
unexplainable  effects. 

The  results  of  the  comparison  are  summarized  In  the  following: 


3.4. 1.; 


Model  Behavior  VJithout  Mechanical  Friction 


Test  A: 


Operation  in  nonlinear  domain.  Fig  B-4. 

The  transition  slope  of  ,  as  retrieved  from  the  plot. 
Is : 


Test  B; 


Test  C; 


=  0.52/0.  172  =  3.02  rad/sec 

Mmax 

which  matches  well  the  theoretical  value  of  3.0  rad/s. 

The  steady  state  value  for  in  the  plot  Is  290  rad/s, 
as  compared  with  theoretically  288  rad/s.  Motor  current 
Ijy  behaves  as  expected,  it  is  proportional  to  The 

magnitude  cannot  easily  be  verified  at  this  stage. 

Operation  at  the  limits  of  linearity.  Fig  B-5.  Is 

just  reaching  Its  saturation  value.  Model  performance 
does  not  show  anv  anomalies  or  unexplainable  effects. 

Operation  In  the  linear  domain,  comparison  with 
analvtlcal  solution  of  Fig  R-9. 

Model  behavior.  Fig  B-6,  matches  verv  well  the  analytical 
solution.  The  model  shows  a  little,  unslgnlf icantly 
higher  overshoot.  Motor  current  cannot  be  compared  since 
It  has  not  been  calculated  in  the  case  of  the  analytical 
solution . 


Test  D:  Nonlinear  operation  with  external  hinge  inonients.  Fig  B-7. 

The  curves  are  Identltn^  with  those  of  Fig  B-4  (lest  A) 
with  the  only  difference  that  1^^  shows  an  offset  of 
2.25  A  in  order  to  compensate  for  the  external  hinge 
moment : 

•  K-  =  2.25  *  0.0707  •  96  -  15.27  Nm 
H  N  M  C 

as  compared  with  the  applied  value  of  15.0  Nm. 

As  long  as  the  motor  current  Is  not  saturated, 
external  loads  do  not  have  any  effect  on  the  dynamic 
behavior  of  the  actuation  system. 

3.4.1.?  Model  Behavior  Including  Mechanical  Friction 

Test  A:  Operation  in  the  nonlinear  domain.  Fig  B-8.  Motor 

current  is  now  required  not  just  during  the  acceleration/ 
deceleration  phases,  but  in  to  overcome  the  friction 
moment  Mp. 

The  current  during  constant  motor  speed,  is  0,3  A,  This 
corresponds  to  a  motor  moment  of 

>  =  •  1,^  =  0,0707  •  0.3  =  0.02121  Nm 

which  matches  verv  well  the  supposed  value  of  M^^  =  0.020. 
The  transition  slope,  is  practically  the  same  as  in  the 
case  without  friction. 

However,  when  the  system  is  approaching  a  steady  state, 
i.e.,  o  t  0,  a  limit  cycle  is  generated,  having 
characteristics  that  are  very  sensitive  to  the  particular 
Implementation.  In  the  present  case  of  a  digital 
Implementation,  frequency  and  amplitude  of  the  limit 
cycle  are  highly  dependent  on  the  particular  Integration 
step  size  chosen.  This  means  that  the  proposed  model  of 
mechanical  friction  is  not  reasonable. 

Since  the  influence  of  friction  on  the  dynamic  behavior 
of  the  actuator  system  is  negligible,  it  is  recommended 
that  the  model  be  operated  with  =  C^,  =  0. 

When  the  model  is  used  within  an  autopilot  loop,  steady 
states  will  practically  never  be  reached  and  the  limit 
cycle  will  probably  never  be  excited.  Therefore,  if  one 
wishes  to  derive  an  estimate  of  power  consumption  of  the 
actuator  system,  the  model  could  be  ur.ed  by  taking 
friction  into  account. 

3.4.2  Eensltivlty  with  Respect  to  Integration  Step  Size 

The  criterion  was  that  the  model  response  should  be  reasonably 
insensitive  with  respect  to  step  slze,PTRK,  as  judged  by  an  expert 
engineer. 


fig  B-10  shcvs  that  tliert-  is  no  significant  difference  between  the 
responses  witli  DTRM  =  0.000'j  s  and  DTRM  =  0.001  s,  whereas  a  cltarlv 
visible  divergence  cap  be  stated  i or  DTRM  =  0.002  s. 

Fig  B-1 1  shows  that  already  T)TRH  =  0.003  s  causes  the  node!  to 
become  unstable. 

Therefore,  the  value  PTRt'  =  0.001  s  appears  to  be  a  reasonable 
choice . 

I’sing  the  same  test  case,  the  model  including  mechaulcal  friction 
was  exercised.  Fig  P-12  shows  the  limit  cycle  in  the  case  of  DTPM  = 
0.001  s.  Fig  B-13  shows  the  results  obtained  with  DTPK  =  0.002  s.  It 
demonstrates  the  sensitivity  of  the  characteristics  of  the  limit  cycle 
with  rer.pect  to  integration  step  size.  This  confirms  the  statement  made 
in  Paragraph  2. 3. 2. 2  about  the  questionable  applicability  of  the  model 
including  mechanical  friction  terms. 

4.  VAMl'ATlON  OF  SYSTFM  MOPFl.’S  STOCHASTIC  COMPONENTS 

The  present  model  was  basically  a  deterministic  nature  in  the  sense 
that  it  does  not  contain  any  internal  sources  of  noise.  Tn  the  real 
syrtvm,  most  of  the  parameters  describing  the  model  are  subject  to 
random  variations  because  of  component  tolerance,  ref  Tab  4.  This  could 
be  taken  into  account  in  the  model  by  random  variation  of  relevant 
parameters  prior  to  each  model  run.  However,  this  feature  has  not  been 
implemented  in  the  present  model. 

5.  VAI.IPATION  ACAIMST  OTHER  PyiSTlNG  MODELS 

No  Validation  against  other  models  have  been  executed, 

6.  SUBSYSTEM  CIIARACTFRIZATION  AND  BRIEF  PFSCPJPTION  OF  SUBSYSTEM 
MOD FES 

Not  applicable, 

7.  BENCHMARK  TEST  CASE 

7 . 1  Description  of  Benchmark  Test  Cases 

The  test  cases  used  for  verification  of  the  dynamic  behavior  of  the 
model  as  described  in  Paragraph  3.3.2.)  should  be  used  as  benchmark  test 
cases. 


7 . 2  Input  Data 

Input  is  a  rectangular  variation  of  the  variable  c  fSIC)  as 
described  in  Paragraph  1.4,1.  ^ 


Output  Data 


The  output  data  is  recorded  in  Figs  B-4  through  B-P,  as  described 
in  Paragraph  1.4,2,  For  the  benchmark  tests,  the  data  base  described  in 
Paragraph  1.6  becomes  the  reference  data  base. 


S( 


7.4 

Criteria  for  Acceptability 

Same 

as  for  verification  tests,  described  in 

Paragraph  3. 

8. 

COMPUTER  PROGRAM 

8.  1 

User  Instructions 

The  model  is  Implemented  as  a  single  subroutine  called  FLACT3.  Its 
calling  sequence  can  readily  be  Inferred  from  the  listing  in  the 
appendix. 

CAUTION: 

The  contents  of  the  state  variables  SI, 
altered  between  calls! 

DSl ,  IC  must  not  be 

For 

Paragraph 

Initlatl zatlon  see  Paragraph  2.2.4  and  for  error  detection  see 

2.2.5. 

00 

Computer  Listings 

The 

complete  source  listing  is  not  included. 

9. 

PROGRAM  VERIFICATION 

9.1 

Criteria  for  Verification 

The  following  criteria  have  been  used  for  program  verification: 

(s)  Correct  Implementation  of  the  mathematical  model  of 
Paragraph  1.6. 

(b)  Program  code  In  compliance  with  Programming  Standards. 

(c)  Portability  of  the  program  code. 

9 . 2  Methods  used  for  Program  Verification 


The  methodology  used  for  model  verification  Is  summarized  In  Tab  8. 
Program  verification  has  been  performed  by  computer  code  analysis. 

Objective  was  to  assure  that  the  model  has  been  correctly 
translated  Into  portable  computer  code. 

Three  types  of  analyses  have  been  performed: 

(a)  Source  Code  Inspection 

(b)  Inspection  of  Cross  Reference  Listing 

(c)  Source  Code  Compilation  In  "ANST-Mode." 


V 


X 


9.3 


Data  Bases  used  for  Progrant  Verification 


9.3.1  Source  Code  Inspection 

Model  Generated  Data  Rase;  Computer  Listing 

Reference  Data  Base:  Description  of  the  mathematical  model  of 
Paragraph  1.6  plus  Programming  Standards,  Paragraph  10,  B-3. 

9.3.3  Inspection  of  Cross  Reference  Listing 

Mode]  Generated  Data  Ease:  Computer  Listing 

Reference  Data  Base:  Requirements  List  of  Paragraph  10,  B-5. 

9.3.3  Source  Code  Compilation  in  "AMST-Mode” 

Model  Generated  Data  Rase;  Diagnostic  messages  by  compilation  run 
in  "ANST-irode" ,  Section  10,  B-6. 

Reference  Data  Base;  AMST-FORTRAM  Standard  as  implemented  in  the 
compiler  of  Section  10,  R-6. 

9.4  Results  from  Program  Verification  Efforts 


9.4.1  Source  Code  Inspection 


Formal  source  code  inspection  was  performed  by  the  Duality 
Assurance  Dept  of  MBB-UA.  Crlte' ion  required  the  compliance  of  the 
computer  listing  with  the  mathematical  model  as  described  in  Paragraph 
1.6  as  well  as  with  the  Programming  Standards  of  Paragraph  10,  B-3. 
This  criterion  was  fulfilled. 

9.4.2  Inspection  of  the  Cross  Reference  Listing 

The  cross  reference  listing,  as  generated  by  the  compiler,  was 
Inspected  by  the  Quality  Assurance  Dept  of  MFB.  It  fulfilled  the 
criterion  to  comply  with  the  requireru'nts  list  of  Section  10,  B-5. 

9.4.3  Source  Code  Compilation  in  "ANSI-Mode" 


Inspection  of  the  source  code  by  the  Quality  Assurance  Dept  of  MBB 
has  shown  that  It  Is  in  compliance  with  Paragraph  10,  B-4.  Compilation 
of  the  program  in  the  "ANST"-mode  did  not  result  In  any  statement 
flagged  as  "non-ANSI". 


10.  APPLICABLE  DOCUMENTS 

B-1  MODEL  DOCUMENTATION,  ELACT3 

ELECTRICAL  ACTUATION  SYSTEM,  CLIMB  2  I.evel  1 
April  19R4 


B-2  REPORT  on  ACCEPTANCE  TESTE  of  the  REAI,  ACTUATOR 


K.l 


B-3  Richtllnier  znr  Programrp.erstel lurp  (Programniing  Standards) 
Internal  MBB-Paper  (in  German) 

B-4  Sandra  Summers,  Jean  Fox 

Writing  Machine  Independent  FORTRAN 
Software  World  Vol  9,  Mo  2 

E-5  Checklist  for  Inspection  of  Cross  Reference  Listing,  as  Generated 
by  CDC-FORTRAN-Compilers. 

Internal  MBB-Paper 

B-6  CnC-FORTRAN-EXTENPED  VERSION  4 
REFERENCE  MANUAL  60997800 

E-7  CPC-NOS  VERSION  1 

REFERENCE  MANUAT.  60475400 


CLIMB  I, EVEN  ?  FPAMPLE 
1.  REA1.  WORT.D  SUBSYSTEM  DATA 

1 . 1  Identification  of  Subsystem 

Electrical  actuator  system  to  drive  the  fins  of  the  experimental 
EMS  missile  system. 

1.2  List  of  Variables  for  Which  Measured  Data  Exist 


Symbol 

— 

Variable 

Model 

Real  System 

a 

commanded  fin  deflection 

c 

RC 

a 

°'rm 

actual  fin  deflection 

'’tm 

m.otor  speed 

1... 

actual  motor  current 

M 

_ 1 

AM 

Attached  to  the  erd  of  CLIMH  Level  3  In  Figs  P-14  through  B-16. 


.S4 

1.3 


2,  EXPERIMENT,  ^-'i  ENVIRONMENT 

2. 1  Scenario  Used  to  Excite  Subsystem 

Laboratory  bench  tests  have  been  performed  with  the  real  actuator 
system  by  applying  step  Inputs.  Test  conditions  A  through  C  as 
identified  Paragraph  3,  B-9,  have  been  used.  Since  nc^  device  was 
available  which  was  capable  to  apply  a  defined  moment  on  the  hinge  under 
dynamic  conditions.  Test  1)  could  not  be  made. 


The  test  experiment  Is  outlined  In  Fig  P-l?.  The  real  actuator 
system  consists  of  two  subassemblies: 

fa)  The  actuator  electronics,  including  the  actuator  controller 
pl:is  the  power  amplifier 

(b)  The  mechanical  parts:  DC-motor,  gear  and  sensors 


Three  power  supplies  were  used  to  feed  the  actuator  electronics: 

2  each  Hewlett  Packard  HP  6012A, 
providing  power  to  the  power  amj)lifier. 

Voltage  setting:  +  ff  V,  precision  ±  0.5  V 
Current  limitation  value:  10  A,  precision  ±  1.0  A 

1  Dual  Power  Supply  Hewlett  Packard  HP  6227  B, 
providing  power  to  the  actuator  controller. 

Voltage  setting:  +  15  V,  precision  t  0.5  V 
Current  limitation:  0.2  A,  precision  ±  5  mA 


The  Input  step  function  to  the  actuator  system  was  provided  bv  an 
EXACT  Function  Generator,  Type  255.  The  control  setting  was; 

Output;  according  to  desired  square  wave  amplitude,  scale 

factor  23.65  V/rad  for  actuator  input 

Frequency:  4  Hz 

-  Waveform:  rectangular 


The  variables  were  recorded  on  a  Gould  Bru.sh  4  Channel  Recorder, 
Type  2400. 


Scale  factors  and  settings  were  as  follows; 


-  Paper  speed:  250  im/s 

-  Recordings: 


Channel  Mo 

Variable 

Scale  Factor 

Channel  Setting 
f.s.  -  25  lines 

1 

”rc 

0.5  V/deg  =  28.65  V/rad 

9.0  V  =  0.312  rad 

2 

°RM 

0.5  V/deg  =  28.65  V/rad 

9.0  V  =  0.312  rad 

3 

'^TM 

11.94  mV/rad  s  ' 

6.0  V  =  500  rad/s 

4 

'am 

1  V/A 

6.25  V  =  6.25  A 

The  points  where  above  variables  have  been  probed  are  depicted  in 
Fig  B-18. 

The  recording  format  corresponds  to  the  one  used  in  the  model 
generated  data  base  for  verification,  see  Paragraph  8,  B-o,  In  order  to 
facilitate  comparisons  by  plot  overlays. 

3.  MPTHOOS  ANP  TF.ClffllQllES  I'SFD  IW  COhl.FCTING  REAL  WORLP  DATA 


3.1  Data  Collection  Methods 


Laboratory  bench  test  with  real  actuator  system,  step  Input  applied 
using  a  square  wave  generator.  Output  recorded  by  a  Brush  4-chanrel 
recorder. 

3. 2  Frror  Sources 

3.2.1  Input  Measurements 


The  apparent  rise  time  on  the  Input  step  function  is  a  function  of 
the  recorder  (bandwidth  approx  50  Hz)  and  not  of  the  generator.  This 
has  been  verified  by  using  an  electronic  scope.  The  accuracy  of  the 
recordings  Is  0.7  percent  of  full  scale. 

3.2.2  Output  Measurements 


3.3 


The  same  conditions  apply. 

Tnput/Output  Data  Analysis 


Input  Data 


3.3. 1 

No  analysis  performed. 

3.3.2  Output  Data 

No  analysis  performed. 

3.4  Contact  for  Further  Information  on  Measured  Data 


Person;  Rudolf  Merz 

Address:  Messerschmitt-Bolkow-Blohm  GmbH 

Abtellung  AE132 
Postfach  801149 
D-8000  Munchen  80 
W-Germany 


Phone  No.:  089-60006536 


4.  VALIDATION  APPROACH 


4.1  Criteria  for  Validation 


The  model  responses  to  various  step  Inputs  should  match  reasonably 
well  the  real  world  data  that  has  been  generated  under  the  same 
conditions.  This  is  Co  be  judged  by  expert  engineers  that  are 
experienced  in  actuator  design  and  missile  system  simulation.  No 
quantitative  measures  for  goodness  of  fit  have  been  used. 

4.2  Validation  Methods  Used 


The  method  of  comparison  of  the  two  date  bases  was  plot  overlays 
and  experts'  judgment. 

4 . 3  Data  Bases  Used  for  Validation 

4.3.1  Reference  Data  Base 


Reference  data  base  is  the  real  world  data  as  shown  In  Figs  B-14 
through  B-16.  The  test  cases  correspond  to  the  ones  described  in 
Paragraph  8,  B-9. 

Test  A;  Operation  in  the  nonlinear  domain.  Fig  B-14. 

Rectangular  Input  =  ±  0.26  rad,  no  external  load. 

The  first  channel  shows  the  step  input.  The  deviation  from  an 
ideal  step  is  due  to  the  limited  bandwidth  of  the  recorder 
(Roughly  50  Hz). 

The  second  channel  shows  actual  fin  deflection.  The  slope  of 
the  ramp  is 

=  0.52/0.168  =  3.095  rad/sec 

nmax 


The  third  charrel  records  the  output  of  the  tacho  generator 
which  serves  as  the  sensor  for  motor  speed.  The  measured 
value  shows  a  ripple  which  Is  due  to  the  cogging  effects  of 
the  motor.  The  frequency  of  the  ripple  is; 

!0  periods/ (21  mm:  200  mm/s)  =  95  Hz, 

Motor  speed  (revolutions  per  second)  is: 

3.095  .  96  =  297.1  rad/s  =  47.3  Hz. 

which  matches  Ideally  taking  into  account  that  the  motor  has  2 
pairs  of  poles. 

The  fourth  channel,  motor  current,  is  shewing  a  large  ripple 
which  is  induced  by  motor  effects  such  as  cogging, 
commutation,  etc.,  as  explained  above. 

Test  B:  Operation  at  the  limits  of  linearity.  Fig  B-15. 

Rectangular  input  =  ±  0.07  rad,  no  external  load. 

Test  C;  Operation  in  the  linear  domain.  Fig  B-16. 

Rectangular  input  =  t  0.017  rad,  no  external  load. 

4.3.2  Model  Generated  Data  Base 

The  model  generated  data  base  used  for  validation  is  documented  in 
Paragraph  8,  E-9. 

4 . 4  Validation  Results 

Test  A;  Operation  in  the  linear  domain. 

Model :  See  B-9,  Paragraph  8. 

Real  System:  Fig  B-14 

The  slope  of  the  ramp  differs  by  2.5%.  The  ripple  on 

top  of  the  variable  ?n^Fig  B-14  is  not  present  in  the  case 
of  the  model  because  the  causing  motor  effects  like 
commutation,  cogging,  etc.  are  not  represented  in  the  model. 

In  the  fourth  channel ,  motor  current  shows  the  greatest 
difference,  whereas,  the  dynamic  behavior  during  the 
acceleration/deceleration  phases  matches  reasonably  well,  a 
A  large  ripple  that  shows  up  is  Induced  by  motor  effects  such 
as  cogging,  commutation,  etc.  that  do  not  exist  In  the  case  of 
the  model  since  motor  effects  are  not  Included. 

Test  B;  Operation  at  the  limits  of  linearity 
Model;  See  B-9,  Paragraph  8. 

Real  System;  Fig  B-15 

As  far  as  o  and  are  concerned,  the  model  is  showing  a 
somewhat  higher  overshoot.  The  correspondence  of  motor 
currents  is  not  very  good  due  to  the  effects  discussed  above. 


4 


i 

( 


ss 


Test  C;  Operatlor  In  the  Unear  domain 
Model:  See  B-9,  Paragraph  8. 

Real  System:  Fig  B-16 

Again,  for  r  and  a  the  match  Is  very  good  with  the  model 
showlrg  a  little  higher  overshoot. 

A  summary  of  validation  results  has  been  given  In  Paragraph  8  of 


4.5  Model  Changes  Due  to  Validation  Effort 

None. 

5.  COMPUTERIZED  EXCITATION  METHODS  USED 

5. 1  Excitation  of  Real  World  System 

The  experiment  used  is  described  In  Paragraph  2.  No  computer  was 
involved . 

5.2  Excitation  of  the  Model 


The  model  was  executed  by  a  main  program  calling  subroutine  ELACT3. 
The  different  test  cases  A  through  C  were  Implemented  by  subsequent 
manual  changes. 

6.  USER  INSTRUCTIONS  FOR  TEST  SET  UP 


No  special  explanations  necessary,  obvious  from  description  of  test 
set  up. 

7.  BENCHMARK  FOR  TEST  SET  UP 

7. 1  Description  of  Benchmark  Test  Cases 

The  test  cases  used  for  the  validation  experiment  as  described  in 
Paragraph  2.1  and  4,3.1  should  be  used  as  benchmark  test  cases. 

7. 2  Initial  Conditions  for  Test  Set  Up 

Ref  Paragraph  6.  The  EXACT  Function  Generator  is  set  up  to 
generate  square  waves  with  a  frequency  of  4  Hz  and  various  amplitudes 
corresponding  to  the  test  cases  A  through  C.  Scale  factor  is 
28.65  V/rad. 

7.3  Results  for  Critical  Variables 

Ref  Paragraph  4.3.1  and  Figs  B-14  through  B-16. 

7.4  Criteria  for  Acceptability  of  Benchmark  Results 

Reasonable  correspondence  with  data  Figs  B-14  through  B-16,  judged 
from  an  engineer's  point  of  view. 


Method  of  comparison;  Plot  overlays. 


8.  APPLICABLE  DOCUMENTS 

B-8  MODEL  DOCUMENTATION,  ELACT3,  ELECTRICAI,  ACTUATION  SYSTEM,  CLIMB 
Level  1 
April  1984 

B-9  dto.,  CLIMB  Level  2 
April  1984 
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Fig  B-8.  Nonlinear  Operation  With  Mechanical  Friction 
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APPENDIX  C 

PROGRAM  DEBUGGING  IN  ACSL 


One  of  the  more  Important  features  of  the  ACSL  language  Is  the 
availability  of  tools  that  assist  In  pinpointing  errors.  The  first 
thing  Is  to  establish  a  frame  of  mind  that  believes  In  the  existence  of 
errors.  It  is  difficult,  in  general,  for  the  average  user  who  writes  a 
model  definition  to  believe  that  there  are  any  errors.  Accept  the  fact 
that  all  programs  have  at  least  one  error  and  part  of  the  joy  of  coming 
up  with  a  finished  product  will  be  In  finding  It, 

As  the  program  is  written,  prepare  the  first  run  for  debugging. 

Set  the  stop  condition  (TERMT)  for  the  first  run  to  a  small  value 
(typically  one  communication  Interval  will  suffice)  so  that  no  time  will 
be  wasted  calculating  the  Incorrect  values.  Use  the  'D'  option  In  the 
translator  so  that  the  program  will  proceed  to  uncover  as  many  errors  as 
possible . 

The  first  run  through  the  translator  will  produce  syntax  error 
Indications  and  probably  error  messages  as  well.  The  translator 
analyzes  each  statement  In  turn  and  if  an  error  occurs  It  will  be 
indicated.  The  way  the  error  is  Indicated  is  to  write  out  again  the 
statement  In  error.  Including  any  continuations,  with  a  line  of 
asterisks  (*)  underneath  to  Indicate  the  acceptable  section.  The 
asterisk  should  stop  just  below  where  the  error  Is  located. 

Example : 


X  =  Y  +  (STN(Y,Y)) 

***SYMTAX  ERROR***THE  LINE  IS  LISTED  WITH  A  POINTER  TO  THE  ERROR 
X  =  Y  +  (SIN(Y,Y)) 

************** 

which  shows  that  the  period  (,)  separating  the  two  Y's  is  not  allowed. 

It  should  be  an  asterisk  (*)  to  indicate  "multiply".  Two  points  should 
be  noted  when  these  errors  are  indicated.  The  first  is  that  only  the 
first  error  in  the  statement  will  be  Indicated,  If  this  error  is 
corrected.  It  may  need  a  second  (or  third)  run  to  uncover  other  problems 
further  Into  the  statement.  IJhen  you  make  a  correction,  take  a  long 
hard  look  at  the  rest  of  the  statement.  The  second  is  that  line  listed 
may  not  look  like  the  input  text  If  continuation  cards  are  used.  The 
error  listing  gives  the  complete  string  to  be  analyzed  after  the 
trailing  blanks  have  been  squeezed  from  the  end  of  any  continued  cards. 

Next,  check  for  misspelling  -  variables  that  should  have  been  the 
same  get  keypunched  wrongly  or  names  that  should  have  been  changed, 
overlooked.  To  check  these,  look  at  the  symbol  cross-reference  tables 
listed  at  the  end  of  the  translator  output.  Any  variables  listed  under 
'VARIABLES  NOT  SPECIFIED  IN  ANY  BLOCK'  will  be  misspellings,  constants 
you  forgot  to  specify,  or  correct  variables  that  had  their  name 
misspelled  at  the  statement  defining  them.  They  should  have  been 
def ined . 
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Next,  take  note  of  any  unsatisfied  external  references  from  the 
load  map.  These  will  usually  correspond  to  arrays  you  forgot  to  declare 
in  an  ARRAY  statement.  Without  this,  they  look  just  like  functions. 

The  first  run-time  command  should  set  up  a  debug  action  with 
usually  the  first  five  or  ten  derivative  evaluations  sufficing.  Include 
the  following  card  at  run-time: 

SET  NDBUG  =  10 

Alternatively,  an  action  can  be  scheduled  that  will  ensure  a  debug 
printout  after  every  START  until  CLEARed: 

ACTION  'VAR'  =  0.0,  'VAL'  =  10,  'LOC  =  NDBUG 

NOTE:  While  the  system  variable  NDBUG  is  greater  than  zero,  the 
complete  set  of  user  variables  Is  printed  out  and  the  the  value  of  NDBUG 
Is  reduced  by  one . 

This  output  is  probably  the  most  Important  data  to  help  In 
debugging;  the  previous  set  of  tools  was  merely  to  ensure  that  the 
mechanics  were  correct  -  commas  In  the  right  place,  spellings 
consistent,  etc.  This  debug  output  gives  the  actual  numbers  calculated 
for  every  one  of  the  state  derivatives  and  Intermediate  variables.  The 
numbers  should  be  examined  carefully  and  checked  for  reasonableness 
using  knowledge  of  the  system  being  modelled.  It  is  a  good  Idea  to 
start  with  initial  conditions  nonzero.  If  there  are  too  many  zero 
values,  the  arithmetic  calculations  can  conceal  errors.  For  preference, 
pick  conditions  so  the  derivatives  all  have  a  nonzero  value  which  can  be 
checked.  Check  the  values  that  are  listed  for  the  constants.  Any  that 
have  been  preset  in  a  CONSTANT  statement  and  where  the  decimal  point  has 
been  left  off  will  be  listed  as  having  a  value  of  0.0.  This  problem  is 
a  very  common  error.  Some  arrays  may  be  missing  from  this  printout  if 
they  happen  to  be  longer  than  the  integer  contained  In  the  system 
variable  MALPRU  (Maximum  Array  Limit  for  Print  Out) .  See  system 
variable  summary  for  the  default  value. 

Now  try  the  first  full  run.  Plan  what  significant  output  variables 
will  yield  correct  model  operation.  Specify  these  In  an  OUTPUT  command. 
Increase  the  termination  time  and  START. 

It  Is  at  this  point  that  the  modeler's  skill  comes  in  to 
rationalize  the  behavior  of  the  simulation  in  terms  of  how  the  real  word 
system  is  expected  to  behave.  About  the  only  help  that  can  be  offered 
is  that  once  questionable  areas  have  been  uncovered,  schedule  debug 
printouts  to  cover  the  area  of  Interest  so  that  as  much  Information  is 
recorded  as  possible.  Note  that  the  debug  output  occurs  after  every 
derivative  evaluation.  For  Runge-Kutta  fourth  order  Integration,  four 
derivative  evaluations  are  made  for  a  time  step  (calculation  Interval): 
one  at  the  beginning,  two  in  the  middle,  and  one  at  the  end.  The 
Independent  variable  will  appear  to  advance  in  half-steps  with  two 
derivative  evaluations  taking  place  each  step.  An  extra  evaluation  will 
take  place  prior  to  each  communication  Interval  or  trip  through  the 
DYNAMIC  section. 
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MEANING  OF  DEBUG  PRINT  OUT 

The  debug  output  Is  generated  by  going  through  the  user  dictionary, 
which  points  to  all  variables  In  the  user  common  block,  and  listing  the 
values  of  each  one  by  one.  The  first  fifteen  variables  are  ACSE  control 
variables  that  are  defined  as  follows: 

la)  T  -  Real;  Independent  variable.  May  have  been  renamed  in  a 
VARIABLE  statement 

(b)  ZZTICG  -  Real;  Initial  condition  on  the  Independent  variable 

(c)  CINT  -  Real;  Current  communication  interval.  May  have  been 
renamed  by  CINTERVAL 

(d)  ZZIFRR  -  Logical;  Variable  step  error  flag.  May  have  been 
renamed  by  ERRTAG 

fe)  ZZNBLK  -  Integer;  Number  of  DERIVATIVE  and  DISCRETE  blocks  In 
use 

(f)  ZZI  -  Integer;  Distinguishes  pre-initial  (=0),  START  (=1)  and 
CONTIN  (=2) 

(g)  ZZST  -  Logical;  Stop  flag  set  by  TERMT  operator 

(h)  ZZFRFL  -  Logical,  First  flag  set  true  at  first  derivative 
evaluation  of  every  step 

(1)  ZZICFL  -  Logical;  Initial  condition  flag  set  true  at  first 

derivative  evaluation  of  every  run  -  immediately  after  Initial 
conditions  have  been  transferred  to  states 

(j)  ZZRNFL  -  Logical;  Reinitialize  flag  set  true  by  REINTT.  Used 
during  initialization  (ZZICFL  =  .TRUE.)  and  then  turned  false 

(k)  ZZNS  -  Integer  array  of  length  number  of  DERIVATIVE  blocks 
giving  number  of  state  variables  in  each  block 

(l)  MINT  -  Real  array  of  length  number  of  DERIVATIVE  blocks  giving 
minimum  integration  step  size  for  each  block.  Name  may  be 
changed  by  global  KINTERVAL  statement 

(m)  MAXT  -  Real  array  of  length  number  of  DERIVATIVE  blocks  giving 
maximum  Integration  step  size  for  each  block.  Name  may  be 
changed  by  global  MAXTERVAl  statement 

(n)  NSTP  -  Integer  array  of  length  number  of  DERIVATIVE  blocks 
giving  communication  Interval  divisor  for  each  block.  Name 
may  be  changed  by  global  NSTF.PS  statement 

(o)  lALG  -  Integer  array  of  length  number  of  DERIVATIVE  blocks 
giving  integration  algorithm  number  to  be  used  for  each  block. 
Name  may  be  changed  by  global  ALGORITHM  statement 
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Next,  In  the  debug  printout  conies  the  list  of  state  variables  In 
DERIVATIVE  block  order  and  In  alphabetical  order  within  each  block,  with 
their  corresponding  derivatives  and  Initial  conditions  on  the  same  line. 
If  line  width  (see  TCWPRN  and  HVDPRN)  Is  sufficient  (126)  the  corre¬ 
sponding  values  of  absolute  error  (XERR)  and  relative  error  (MERR)  are 
also  listed  on  the  same  line.  In  general,  the  derivatives  will  all  be 
dummy  variables  (ZOnnnn  form)  except  for  those  defined  by  the  INTVC 
Integration  operator. 

All  the  algebraic  variables  follow  the  states  In  alphabetical 
order.  Any  EOUIVALENCED  variables  are  listed  at  the  end.  System 
variable  ZZSEED  contains  the  random  number  seed  variable  which  will 
change  (depends  on  machine  type)  with  every  call  for  a  new  random 
number.  ZZTLXP  is  a  logical  variable  present  In  some  machine  versions 
to  request  the  reprleve/lnterrupt  capability.  If  It  Is  set  false  before 
the  first  START,  normal  system  dumps  can  be  obtained  if  desired. 


DEBUG 

A  call  to  this  routine  will  produce  a  debug  list  of  all  variables, 
excluding  arrays  greater  than  MALPRN  (Maximum  Array  Limit  for  Print)  on 
both  PRN  and  DIS  units.  Tlie  technique  of  setting  NDPUG  to  a  positive 
Integer;  yields  a  debug  list  at  the  end  of  every  derivative  evaluation. 
While  useful  as  a  checkout  tool  with  large  programs,  this  action  can 
produce  an  overwhelming  amount  of  output.  Selective  output  can  now  be 
obtained  by: 

IF  (logical  condition)  CALL  DEBUG 
Included  In  the  DERIVATIVE  section.  Including  the  statement 
CALL  DEBUG 

In  the  DYNAMIC  section  produces  the  entire  list  at  each  communication 
interval  and  Is  synonymous  with  asking  for  the  OUTPUT  of  all  variables. 

Including 

IF  (DUMP)  CALL  DEBUG 

In  the  TERMINAL  section  Is  a  useful  artifice  since  all  final  values  are 
displayed  as  well  a.s  the  Initial  conditions  for  that  run. 
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1 4.  Abstract 

A  survey  of  missile  simulation  facilities  m  1 982  indicated  that  very  tittle  has  been  devoted  It)  the 
validation  of  missile  system  simulations.  Those  validation  techniques  that  are  used  are  not 
standardized,  are  often  ill  defined  and  are  generally  undocumented.  In  an  attempt  to  rectify  this 
situation,  the  AGARD  Flight  Mechanics  Panel  established  Working  Group  1 2  to  examine  missile 
simulation  procedures  and  recommend  techniques  for  the  validation  and  universalization  of  the 
results  of  such  procedures.  This  report  presents  the  output  of  this  Working  Group. 

The  report  recommends  a  simulation  terminology  lhayf  adopted,  should  simplify  the  validation 
processs.  A  hierarchical  model  repres^entation  called.  “Confidence  Level  in  Model  Behaviour," 
(CLIMB)  is  also  recommended  tk^organizes  the  simulation  process  and  emphasizes  a  standardized 
documentation  procedure  for  both  simplifying  the  orderly  use  of  simulation  and  insuring  confidence 
in  the  final  simulation  results.  Examples  of  using  the  CLIMB  model  are  included  in  the  Appendices. 

A  brief  section  on  Computer  Languages  and  how  they  can  benefit  the  simulation  process  is  included. 
Software  verification,  validation  and  assessment  methods  are  examii/ed  and  recommendations  made 
to  enhance  the  validation  of  the  simulation,  r  '  ' 
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This  report  repre,sents  the  combined  efforts  of  a  group  consisting  of  representatives  from  France, 
West  Germany.  Italy.  United  Kingdom  and  United  Stales.  The  information  pre.senled.  therefore,  is 
truly  international  in  nature. 

This  Advisory  Report  was  prepared  at  the  request  of  the  AGARD  Flight  Mechanics  Panel. 
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